Fair Value - Financial Assets and Liabilities (Details) - USD ($) $ in Thousands	Mar. 31, 2016	Dec. 31, 2015
Assets and liabilities measured at fair value
Marketable securities	$ 649,364	$ 724,947
Domestic servicing rights	95,492	119,698
Derivative assets	36,938	45,091
VIE Assets	85,115,662	76,675,689
Derivative liabilities	16,202	5,196
VIE Liabilities	84,151,022	75,817,014
Fair value measurements on recurring basis
Assets and liabilities measured at fair value
Loans held-for-sale, fair value option	154,225	203,865
Derivative assets	36,938	45,091
VIE Assets	85,115,662	76,675,689
Total	85,723,850	77,448,046
Derivative liabilities	16,202	5,196
VIE Liabilities	84,151,022	75,817,014
Total	84,167,224	75,822,210
Fair value measurements on recurring basis | Domestic servicing rights
Assets and liabilities measured at fair value
Domestic servicing rights	95,492	119,698
Fair value measurements on recurring basis | RMBS
Assets and liabilities measured at fair value
Available-for-sale securities	210,898	176,224
Fair value measurements on recurring basis | CMBS
Assets and liabilities measured at fair value
Available-for-sale securities	96,724	212,981
Fair value measurements on recurring basis | Equity security
Assets and liabilities measured at fair value
Marketable securities	13,911	14,498
Fair value measurements on recurring basis | Level I
Assets and liabilities measured at fair value
Total	13,911	14,498
Fair value measurements on recurring basis | Level I | Equity security
Assets and liabilities measured at fair value
Marketable securities	13,911	14,498
Fair value measurements on recurring basis | Level II
Assets and liabilities measured at fair value
Derivative assets	36,938	45,091
Total	36,938	45,091
Derivative liabilities	16,202	5,196
VIE Liabilities	81,112,488	73,264,566
Total	81,128,690	73,269,762
Fair value measurements on recurring basis | Level III
Assets and liabilities measured at fair value
Loans held-for-sale, fair value option	154,225	203,865
VIE Assets	85,115,662	76,675,689
Total	85,673,001	77,388,457
VIE Liabilities	3,038,534	2,552,448
Total	3,038,534	2,552,448
Fair value measurements on recurring basis | Level III | Domestic servicing rights
Assets and liabilities measured at fair value
Domestic servicing rights	95,492	119,698
Fair value measurements on recurring basis | Level III | RMBS
Assets and liabilities measured at fair value
Available-for-sale securities	210,898	176,224
Fair value measurements on recurring basis | Level III | CMBS
Assets and liabilities measured at fair value
Available-for-sale securities	$ 96,724	$ 212,981