Fair Value - Financial Assets and Liabilities (Details) - USD ($) $ in Thousands	Jun. 30, 2016	Dec. 31, 2015
Assets and liabilities measured at fair value
Marketable securities	$ 898,803	$ 724,947
Domestic servicing rights	83,301	119,698
Derivative assets	42,692	45,091
VIE Assets	80,076,117	76,675,689
Derivative liabilities	17,870	5,196
VIE Liabilities	79,087,142	75,817,014
Fair value measurements on recurring basis
Assets and liabilities measured at fair value
Loans held-for-sale, fair value option	237,106	203,865
Derivative assets	42,692	45,091
VIE Assets	80,076,117	76,675,689
Total	80,817,677	77,448,046
Derivative liabilities	17,870	5,196
VIE Liabilities	79,087,142	75,817,014
Total	79,105,012	75,822,210
Fair value measurements on recurring basis | Domestic servicing rights
Assets and liabilities measured at fair value
Domestic servicing rights	83,301	119,698
Fair value measurements on recurring basis | RMBS
Assets and liabilities measured at fair value
Available-for-sale securities	251,260	176,224
Fair value measurements on recurring basis | CMBS
Assets and liabilities measured at fair value
Available-for-sale securities	114,340	212,981
Fair value measurements on recurring basis | Equity security
Assets and liabilities measured at fair value
Marketable securities	12,861	14,498
Fair value measurements on recurring basis | Level I
Assets and liabilities measured at fair value
Total	12,861	14,498
Fair value measurements on recurring basis | Level I | Equity security
Assets and liabilities measured at fair value
Marketable securities	12,861	14,498
Fair value measurements on recurring basis | Level II
Assets and liabilities measured at fair value
Derivative assets	42,692	45,091
Total	42,692	45,091
Derivative liabilities	17,870	5,196
VIE Liabilities	75,546,490	73,264,566
Total	75,564,360	73,269,762
Fair value measurements on recurring basis | Level III
Assets and liabilities measured at fair value
Loans held-for-sale, fair value option	237,106	203,865
VIE Assets	80,076,117	76,675,689
Total	80,762,124	77,388,457
VIE Liabilities	3,540,652	2,552,448
Total	3,540,652	2,552,448
Fair value measurements on recurring basis | Level III | Domestic servicing rights
Assets and liabilities measured at fair value
Domestic servicing rights	83,301	119,698
Fair value measurements on recurring basis | Level III | RMBS
Assets and liabilities measured at fair value
Available-for-sale securities	251,260	176,224
Fair value measurements on recurring basis | Level III | CMBS
Assets and liabilities measured at fair value
Available-for-sale securities	$ 114,340	$ 212,981