Fair Value - Financial Assets and Liabilities (Details) - USD ($) $ in Thousands	Sep. 30, 2017	Dec. 31, 2016
Assets and liabilities measured at fair value
Permitted reinvestment under static investment in VIEs	$ 0
Marketable securities	701,818	$ 807,618
Domestic servicing rights	33,781	55,082
Derivative assets	38,293	89,361
VIE Assets	51,197,981	67,123,261
Derivative liabilities	22,890	3,904
VIE Liabilities	50,150,781	66,130,592
Fair value measurements on recurring basis
Assets and liabilities measured at fair value
Loans held-for-sale, fair value option	608,624	63,279
Derivative assets	38,293	89,361
VIE Assets	51,197,981	67,123,261
Total	52,169,301	67,628,621
Derivative liabilities	22,890	3,904
VIE Liabilities	50,150,781	66,130,592
Total	50,173,671	66,134,496
Fair value measurements on recurring basis | Domestic servicing rights
Assets and liabilities measured at fair value
Domestic servicing rights	33,781	55,082
Fair value measurements on recurring basis | RMBS
Assets and liabilities measured at fair value
Available-for-sale securities	253,252	253,915
Fair value measurements on recurring basis | CMBS
Assets and liabilities measured at fair value
Available-for-sale securities	23,841	31,546
Fair value measurements on recurring basis | Equity security
Assets and liabilities measured at fair value
Marketable securities	13,529	12,177
Fair value measurements on recurring basis | Level I
Assets and liabilities measured at fair value
Total	13,529	12,177
Fair value measurements on recurring basis | Level I | Equity security
Assets and liabilities measured at fair value
Marketable securities	13,529	12,177
Fair value measurements on recurring basis | Level II
Assets and liabilities measured at fair value
Derivative assets	38,293	89,361
Total	38,293	89,361
Derivative liabilities	22,890	3,904
VIE Liabilities	47,890,998	63,545,223
Total	47,913,888	63,549,127
Fair value measurements on recurring basis | Level III
Assets and liabilities measured at fair value
Loans held-for-sale, fair value option	608,624	63,279
VIE Assets	51,197,981	67,123,261
Total	52,117,479	67,527,083
VIE Liabilities	2,259,783	2,585,369
Total	2,259,783	2,585,369
Fair value measurements on recurring basis | Level III | Domestic servicing rights
Assets and liabilities measured at fair value
Domestic servicing rights	33,781	55,082
Fair value measurements on recurring basis | Level III | RMBS
Assets and liabilities measured at fair value
Available-for-sale securities	253,252	253,915
Fair value measurements on recurring basis | Level III | CMBS
Assets and liabilities measured at fair value
Available-for-sale securities	$ 23,841	$ 31,546