Fair Value - Financial Assets and Liabilities (Details) - USD ($) $ in Thousands	Sep. 30, 2021	Dec. 31, 2020
Financial Assets:
Available-for-sale securities	$ 148,583	$ 167,349
Domestic servicing rights	15,942	13,202
Derivative assets	52,566	40,555
Assets	81,980,864	80,873,509
Liabilities, Fair Value Disclosure [Abstract]
Derivative liabilities	15,615	41,324
Liabilities	77,133,752	76,010,933
Fair value measurements on recurring basis
Financial Assets:
Equity security, fair value	12,067	11,247
Derivative assets	52,566	40,555
Total Assets	64,789,906	65,513,117
Liabilities, Fair Value Disclosure [Abstract]
Derivative liabilities	15,615	41,324
Total Liabilities	60,910,590	62,817,695
Fair value measurements on recurring basis | Loans held-for-sale
Financial Assets:
Loans under fair value option	2,190,765	1,022,979
Fair value measurements on recurring basis | RMBS
Financial Assets:
Available-for-sale securities	148,583	167,349
Fair value measurements on recurring basis | CMBS
Financial Assets:
Available-for-sale securities	23,503	19,457
Fair value measurements on recurring basis | Domestic servicing rights, at fair value
Financial Assets:
Domestic servicing rights	15,942	13,202
Fair value measurements on recurring basis | Level I
Financial Assets:
Equity security, fair value	12,067	11,247
Derivative assets	0	0
Total Assets	12,067	11,247
Liabilities, Fair Value Disclosure [Abstract]
Derivative liabilities	0	0
Total Liabilities	0	0
Fair value measurements on recurring basis | Level I | Loans held-for-sale
Financial Assets:
Loans under fair value option	0	0
Fair value measurements on recurring basis | Level I | RMBS
Financial Assets:
Available-for-sale securities	0	0
Fair value measurements on recurring basis | Level I | CMBS
Financial Assets:
Available-for-sale securities	0	0
Fair value measurements on recurring basis | Level I | Domestic servicing rights, at fair value
Financial Assets:
Domestic servicing rights	0	0
Fair value measurements on recurring basis | Level II
Financial Assets:
Equity security, fair value	0	0
Derivative assets	52,566	40,555
Total Assets	437,115	40,555
Liabilities, Fair Value Disclosure [Abstract]
Derivative liabilities	15,615	41,324
Total Liabilities	56,380,001	60,797,819
Fair value measurements on recurring basis | Level II | Loans held-for-sale
Financial Assets:
Loans under fair value option	384,549	0
Fair value measurements on recurring basis | Level II | RMBS
Financial Assets:
Available-for-sale securities	0	0
Fair value measurements on recurring basis | Level II | CMBS
Financial Assets:
Available-for-sale securities	0	0
Fair value measurements on recurring basis | Level II | Domestic servicing rights, at fair value
Financial Assets:
Domestic servicing rights	0	0
Fair value measurements on recurring basis | Level III
Financial Assets:
Equity security, fair value	0	0
Derivative assets	0	0
Total Assets	64,340,724	65,461,315
Liabilities, Fair Value Disclosure [Abstract]
Derivative liabilities	0	0
Total Liabilities	4,530,589	2,019,876
Fair value measurements on recurring basis | Level III | Loans held-for-sale
Financial Assets:
Loans under fair value option	1,806,216	1,022,979
Fair value measurements on recurring basis | Level III | RMBS
Financial Assets:
Available-for-sale securities	148,583	167,349
Fair value measurements on recurring basis | Level III | CMBS
Financial Assets:
Available-for-sale securities	23,503	19,457
Fair value measurements on recurring basis | Level III | Domestic servicing rights, at fair value
Financial Assets:
Domestic servicing rights	15,942	13,202
Primary beneficiary | Fair value measurements on recurring basis | VIE assets
Financial Assets:
Assets	62,346,480	64,238,328
Primary beneficiary | Fair value measurements on recurring basis | VIE liabilities
Liabilities, Fair Value Disclosure [Abstract]
Liabilities	60,894,975	62,776,371
Primary beneficiary | Fair value measurements on recurring basis | Level I | VIE assets
Financial Assets:
Assets	0	0
Primary beneficiary | Fair value measurements on recurring basis | Level I | VIE liabilities
Liabilities, Fair Value Disclosure [Abstract]
Liabilities	0	0
Primary beneficiary | Fair value measurements on recurring basis | Level II | VIE assets
Financial Assets:
Assets	0	0
Primary beneficiary | Fair value measurements on recurring basis | Level II | VIE liabilities
Liabilities, Fair Value Disclosure [Abstract]
Liabilities	56,364,386	60,756,495
Primary beneficiary | Fair value measurements on recurring basis | Level III | VIE assets
Financial Assets:
Assets	62,346,480	64,238,328
Primary beneficiary | Fair value measurements on recurring basis | Level III | VIE liabilities
Liabilities, Fair Value Disclosure [Abstract]
Liabilities	$ 4,530,589	$ 2,019,876