Fair Value - Financial Assets and Liabilities (Details) - USD ($) $ in Thousands	Mar. 31, 2021	Dec. 31, 2020
Assets and liabilities measured at fair value
Permitted reinvestment under static investment in VIEs	$ 0
Marketable securities	678,287	$ 736,658
Domestic servicing rights	12,406	13,202
Derivative assets	38,029	40,555
Total Assets	79,702,323	80,873,509
Total Liabilities	74,847,538	76,010,933
Derivative liabilities	34,805	41,324
Fair value measurements on recurring basis
Assets and liabilities measured at fair value
Derivative assets	38,029	40,555
Total Assets	63,371,530	65,513,117
Derivative liabilities	34,805	41,324
Total Liabilities	60,931,514	62,817,695
Fair value measurements on recurring basis | Loans held-for-sale
Assets and liabilities measured at fair value
Loans held-for-sale, fair value option	763,773	1,022,979
Fair value measurements on recurring basis | RMBS
Assets and liabilities measured at fair value
Available-for-sale securities	160,301	167,349
Fair value measurements on recurring basis | CMBS
Assets and liabilities measured at fair value
Available-for-sale securities	19,256	19,457
Fair value measurements on recurring basis | Equity security
Assets and liabilities measured at fair value
Marketable securities	10,655	11,247
Fair value measurements on recurring basis | Domestic Servicing Rights
Assets and liabilities measured at fair value
Domestic servicing rights	12,406	13,202
Fair value measurements on recurring basis | Level I
Assets and liabilities measured at fair value
Total Assets	10,655	11,247
Fair value measurements on recurring basis | Level I | Equity security
Assets and liabilities measured at fair value
Marketable securities	10,655	11,247
Fair value measurements on recurring basis | Level II
Assets and liabilities measured at fair value
Derivative assets	38,029	40,555
Total Assets	38,029	40,555
Derivative liabilities	34,805	41,324
Total Liabilities	58,704,086	60,797,819
Fair value measurements on recurring basis | Level III
Assets and liabilities measured at fair value
Total Assets	63,322,846	65,461,315
Total Liabilities	2,227,428	2,019,876
Fair value measurements on recurring basis | Level III | Loans held-for-sale
Assets and liabilities measured at fair value
Loans held-for-sale, fair value option	763,773	1,022,979
Fair value measurements on recurring basis | Level III | RMBS
Assets and liabilities measured at fair value
Available-for-sale securities	160,301	167,349
Fair value measurements on recurring basis | Level III | CMBS
Assets and liabilities measured at fair value
Available-for-sale securities	19,256	19,457
Fair value measurements on recurring basis | Level III | Domestic Servicing Rights
Assets and liabilities measured at fair value
Domestic servicing rights	12,406	13,202
Primary beneficiary
Assets and liabilities measured at fair value
Total Assets	62,367,110	64,238,328
Total Liabilities	60,896,709	62,776,371
Primary beneficiary | Fair value measurements on recurring basis | VIE Assets
Assets and liabilities measured at fair value
Total Assets	62,367,110	64,238,328
Primary beneficiary | Fair value measurements on recurring basis | VIE liabilities
Assets and liabilities measured at fair value
Total Liabilities	60,896,709	62,776,371
Primary beneficiary | Fair value measurements on recurring basis | Level II | VIE liabilities
Assets and liabilities measured at fair value
Total Liabilities	58,669,281	60,756,495
Primary beneficiary | Fair value measurements on recurring basis | Level III | VIE Assets
Assets and liabilities measured at fair value
Total Assets	62,367,110	64,238,328
Primary beneficiary | Fair value measurements on recurring basis | Level III | VIE liabilities
Assets and liabilities measured at fair value
Total Liabilities	$ 2,227,428	$ 2,019,876