Fair Value - Financial Assets and Liabilities (Details) - USD ($) $ in Thousands	Dec. 31, 2016	Dec. 31, 2015
Assets and liabilities measured at fair value
Permitted reinvestment under static investment in VIEs	$ 0
Marketable securities	807,618	$ 724,947
Domestic servicing rights	55,082	119,698
Derivative assets	89,361	45,091
VIE Assets	67,123,261	76,675,689
Derivative liabilities	3,904	5,196
VIE Liabilities	66,130,592	75,817,014
Fair value measurements on recurring basis
Assets and liabilities measured at fair value
Loans held-for-sale, fair value option	63,279	203,865
Derivative assets	89,361	45,091
VIE Assets	67,123,261	76,675,689
Total	67,628,621	77,448,046
Derivative liabilities	3,904	5,196
VIE Liabilities	66,130,592	75,817,014
Total	66,134,496	75,822,210
Fair value measurements on recurring basis | Domestic servicing rights
Assets and liabilities measured at fair value
Domestic servicing rights	55,082	119,698
Fair value measurements on recurring basis | RMBS
Assets and liabilities measured at fair value
Available-for-sale securities	253,915	176,224
Fair value measurements on recurring basis | CMBS
Assets and liabilities measured at fair value
Available-for-sale securities	31,546	212,981
Fair value measurements on recurring basis | Equity security
Assets and liabilities measured at fair value
Marketable securities	12,177	14,498
Fair value measurements on recurring basis | Level I
Assets and liabilities measured at fair value
Total	12,177	14,498
Fair value measurements on recurring basis | Level I | Equity security
Assets and liabilities measured at fair value
Marketable securities	12,177	14,498
Fair value measurements on recurring basis | Level II
Assets and liabilities measured at fair value
Derivative assets	89,361	45,091
Total	89,361	45,091
Derivative liabilities	3,904	5,196
VIE Liabilities	63,545,223	73,264,566
Total	63,549,127	73,269,762
Fair value measurements on recurring basis | Level III
Assets and liabilities measured at fair value
Loans held-for-sale, fair value option	63,279	203,865
VIE Assets	67,123,261	76,675,689
Total	67,527,083	77,388,457
VIE Liabilities	2,585,369	2,552,448
Total	2,585,369	2,552,448
Fair value measurements on recurring basis | Level III | Domestic servicing rights
Assets and liabilities measured at fair value
Domestic servicing rights	55,082	119,698
Fair value measurements on recurring basis | Level III | RMBS
Assets and liabilities measured at fair value
Available-for-sale securities	253,915	176,224
Fair value measurements on recurring basis | Level III | CMBS
Assets and liabilities measured at fair value
Available-for-sale securities	$ 31,546	$ 212,981