Fair Value - Financial Assets and Liabilities (Details) - USD ($) $ in Thousands	Dec. 31, 2020	Dec. 31, 2019
Assets and liabilities measured at fair value
Permitted reinvestment under static investment in VIEs	$ 0
Marketable securities	736,658	$ 810,238
Domestic servicing rights	13,202	16,917
Derivative assets	40,555	28,943
Total Assets	80,873,509	78,042,336
Total Liabilities	76,010,933	72,905,322
Derivative liabilities	41,324	8,740
Fair value measurements on recurring basis
Assets and liabilities measured at fair value
Derivative assets		28,943
Total Assets	65,513,117	63,908,829
Derivative liabilities	41,324	8,740
Total Liabilities	62,817,695	60,752,234
Fair value measurements on recurring basis | Loans held-for-sale
Assets and liabilities measured at fair value
Loans held-for-sale, fair value option	1,022,979	1,436,194
Fair value measurements on recurring basis | RMBS
Assets and liabilities measured at fair value
Available-for-sale securities	167,349	189,576
Fair value measurements on recurring basis | CMBS
Assets and liabilities measured at fair value
Available-for-sale securities	19,457	37,360
Fair value measurements on recurring basis | Equity security
Assets and liabilities measured at fair value
Marketable securities	11,247	12,664
Fair value measurements on recurring basis | Domestic Servicing Rights
Assets and liabilities measured at fair value
Domestic servicing rights	13,202	16,917
Fair value measurements on recurring basis | Level I
Assets and liabilities measured at fair value
Total Assets	11,247	12,664
Fair value measurements on recurring basis | Level I | Equity security
Assets and liabilities measured at fair value
Marketable securities	11,247	12,664
Fair value measurements on recurring basis | Level II
Assets and liabilities measured at fair value
Derivative assets		28,943
Total Assets	40,555	41,295
Derivative liabilities	41,324	8,740
Total Liabilities	60,797,819	58,214,842
Fair value measurements on recurring basis | Level II | CMBS
Assets and liabilities measured at fair value
Available-for-sale securities		12,352
Fair value measurements on recurring basis | Level III
Assets and liabilities measured at fair value
Total Assets	65,461,315	63,854,870
Total Liabilities	2,019,876	2,537,392
Fair value measurements on recurring basis | Level III | Loans held-for-sale
Assets and liabilities measured at fair value
Loans held-for-sale, fair value option	1,022,979	1,436,194
Fair value measurements on recurring basis | Level III | RMBS
Assets and liabilities measured at fair value
Available-for-sale securities	167,349	189,576
Fair value measurements on recurring basis | Level III | CMBS
Assets and liabilities measured at fair value
Available-for-sale securities	19,457	25,008
Fair value measurements on recurring basis | Level III | Domestic Servicing Rights
Assets and liabilities measured at fair value
Domestic servicing rights	13,202	16,917
Primary beneficiary
Assets and liabilities measured at fair value
Total Assets	64,238,328	62,187,175
Total Liabilities	62,776,371	60,743,494
Primary beneficiary | Fair value measurements on recurring basis
Assets and liabilities measured at fair value
Derivative assets	40,555
Total Assets	64,238,328	62,187,175
Total Liabilities	62,776,371	60,743,494
Primary beneficiary | Fair value measurements on recurring basis | Level II
Assets and liabilities measured at fair value
Derivative assets	40,555
Total Liabilities	60,756,495	58,206,102
Primary beneficiary | Fair value measurements on recurring basis | Level III
Assets and liabilities measured at fair value
Total Assets	64,238,328	62,187,175
Total Liabilities	2,019,876	$ 2,537,392
Primary beneficiary | Fair value measurements on recurring basis | Level III | VIE Assets
Assets and liabilities measured at fair value
Total Assets	64,238,328
Primary beneficiary | Fair value measurements on recurring basis | Level III | VIE liabilities
Assets and liabilities measured at fair value
Total Liabilities	$ 2,019,876