Fair Value - Financial Assets and Liabilities (Details) - USD ($) $ in Thousands	Jun. 30, 2020	Dec. 31, 2019
Assets and liabilities measured at fair value
Permitted reinvestment under static investment in VIEs	$ 0
Marketable securities	752,025	$ 810,238
Domestic servicing rights	13,955	16,917
Derivative assets	90,905	28,943
Total Assets	79,556,279	78,042,336
Total Liabilities	74,691,483	72,905,322
Derivative liabilities	3,880	8,740
Fair value measurements on recurring basis
Assets and liabilities measured at fair value
Derivative assets	90,905	28,943
Total Assets	65,382,462	63,908,829
Derivative liabilities	3,880	8,740
Total Liabilities	62,621,855	60,752,234
Fair value measurements on recurring basis | Loans held-for-sale
Assets and liabilities measured at fair value
Loans held-for-sale, fair value option	894,613	1,436,194
Fair value measurements on recurring basis | RMBS
Assets and liabilities measured at fair value
Available-for-sale securities	174,281	189,576
Fair value measurements on recurring basis | CMBS
Assets and liabilities measured at fair value
Available-for-sale securities	23,530	37,360
Fair value measurements on recurring basis | Equity security
Assets and liabilities measured at fair value
Marketable securities	9,791	12,664
Fair value measurements on recurring basis | Domestic Servicing Rights
Assets and liabilities measured at fair value
Domestic servicing rights	13,955	16,917
Fair value measurements on recurring basis | Level I
Assets and liabilities measured at fair value
Total Assets	9,791	12,664
Fair value measurements on recurring basis | Level I | Equity security
Assets and liabilities measured at fair value
Marketable securities	9,791	12,664
Fair value measurements on recurring basis | Level II
Assets and liabilities measured at fair value
Derivative assets	90,905	28,943
Total Assets	92,544	41,295
Derivative liabilities	3,880	8,740
Total Liabilities	60,492,326	58,214,842
Fair value measurements on recurring basis | Level II | CMBS
Assets and liabilities measured at fair value
Available-for-sale securities	1,639	12,352
Fair value measurements on recurring basis | Level III
Assets and liabilities measured at fair value
Total Assets	65,280,127	63,854,870
Total Liabilities	2,129,529	2,537,392
Fair value measurements on recurring basis | Level III | Loans held-for-sale
Assets and liabilities measured at fair value
Loans held-for-sale, fair value option	894,613	1,436,194
Fair value measurements on recurring basis | Level III | RMBS
Assets and liabilities measured at fair value
Available-for-sale securities	174,281	189,576
Fair value measurements on recurring basis | Level III | CMBS
Assets and liabilities measured at fair value
Available-for-sale securities	21,891	25,008
Fair value measurements on recurring basis | Level III | Domestic Servicing Rights
Assets and liabilities measured at fair value
Domestic servicing rights	13,955	16,917
Primary beneficiary
Assets and liabilities measured at fair value
Total Assets	64,175,387	62,187,175
Total Liabilities	62,617,975	60,743,494
Primary beneficiary | Fair value measurements on recurring basis | VIE Assets
Assets and liabilities measured at fair value
Total Assets	64,175,387	62,187,175
Primary beneficiary | Fair value measurements on recurring basis | VIE liabilities
Assets and liabilities measured at fair value
Total Liabilities	62,617,975	60,743,494
Primary beneficiary | Fair value measurements on recurring basis | Level II | VIE liabilities
Assets and liabilities measured at fair value
Total Liabilities	60,488,446	58,206,102
Primary beneficiary | Fair value measurements on recurring basis | Level III | VIE Assets
Assets and liabilities measured at fair value
Total Assets	64,175,387	62,187,175
Primary beneficiary | Fair value measurements on recurring basis | Level III | VIE liabilities
Assets and liabilities measured at fair value
Total Liabilities	$ 2,129,529	$ 2,537,392