Fair Value - Financial Assets and Liabilities (Details) - USD ($) $ in Thousands	Mar. 31, 2017	Dec. 31, 2016
Assets and liabilities measured at fair value
Permitted reinvestment under static investment in VIEs	$ 0
Marketable securities	730,171	$ 807,618
Domestic servicing rights	46,649	55,082
Derivative assets	71,040	89,361
VIE Assets	60,185,851	67,123,261
Derivative liabilities	1,567	3,904
VIE Liabilities	59,147,068	66,130,592
Fair value measurements on recurring basis
Assets and liabilities measured at fair value
Loans held-for-sale, fair value option	340,266	63,279
Derivative assets	71,040	89,361
VIE Assets	60,185,851	67,123,261
Total	60,921,252	67,628,621
Derivative liabilities	1,567	3,904
VIE Liabilities	59,147,068	66,130,592
Total	59,148,635	66,134,496
Fair value measurements on recurring basis | Domestic servicing rights
Assets and liabilities measured at fair value
Domestic servicing rights	46,649	55,082
Fair value measurements on recurring basis | RMBS
Assets and liabilities measured at fair value
Available-for-sale securities	249,419	253,915
Fair value measurements on recurring basis | CMBS
Assets and liabilities measured at fair value
Available-for-sale securities	15,472	31,546
Fair value measurements on recurring basis | Equity security
Assets and liabilities measured at fair value
Marketable securities	12,555	12,177
Fair value measurements on recurring basis | Level I
Assets and liabilities measured at fair value
Total	12,555	12,177
Fair value measurements on recurring basis | Level I | Equity security
Assets and liabilities measured at fair value
Marketable securities	12,555	12,177
Fair value measurements on recurring basis | Level II
Assets and liabilities measured at fair value
Derivative assets	71,040	89,361
Total	71,040	89,361
Derivative liabilities	1,567	3,904
VIE Liabilities	56,985,773	63,545,223
Total	56,987,340	63,549,127
Fair value measurements on recurring basis | Level III
Assets and liabilities measured at fair value
Loans held-for-sale, fair value option	340,266	63,279
VIE Assets	60,185,851	67,123,261
Total	60,837,657	67,527,083
VIE Liabilities	2,161,295	2,585,369
Total	2,161,295	2,585,369
Fair value measurements on recurring basis | Level III | Domestic servicing rights
Assets and liabilities measured at fair value
Domestic servicing rights	46,649	55,082
Fair value measurements on recurring basis | Level III | RMBS
Assets and liabilities measured at fair value
Available-for-sale securities	249,419	253,915
Fair value measurements on recurring basis | Level III | CMBS
Assets and liabilities measured at fair value
Available-for-sale securities	$ 15,472	$ 31,546