Fair Value - Financial Assets and Liabilities (Details) - USD ($) $ in Thousands	Jun. 30, 2021	Dec. 31, 2020
Financial Assets:
Available-for-sale securities	$ 154,704	$ 167,349
Domestic servicing rights	13,705	13,202
Derivative assets	35,615	40,555
Assets	81,309,302	80,873,509
Liabilities, Fair Value Disclosure [Abstract]
Derivative liabilities	33,061	41,324
Liabilities	76,457,602	76,010,933
Fair value measurements on recurring basis
Financial Assets:
Equity security, fair value	11,880	11,247
Derivative assets	35,615	40,555
Total Assets	64,747,173	65,513,117
Liabilities, Fair Value Disclosure [Abstract]
Derivative liabilities	33,061	41,324
Total Liabilities	62,034,771	62,817,695
Fair value measurements on recurring basis | Loans held-for-sale
Financial Assets:
Loans under fair value option	1,018,508	1,022,979
Fair value measurements on recurring basis | RMBS
Financial Assets:
Available-for-sale securities	154,704	167,349
Fair value measurements on recurring basis | CMBS
Financial Assets:
Available-for-sale securities	18,965	19,457
Fair value measurements on recurring basis | Domestic servicing rights, at fair value
Financial Assets:
Domestic servicing rights	13,705	13,202
Fair value measurements on recurring basis | Level I
Financial Assets:
Equity security, fair value	11,880	11,247
Derivative assets	0	0
Total Assets	11,880	11,247
Liabilities, Fair Value Disclosure [Abstract]
Derivative liabilities	0	0
Total Liabilities	0	0
Fair value measurements on recurring basis | Level I | Loans held-for-sale
Financial Assets:
Loans under fair value option	0	0
Fair value measurements on recurring basis | Level I | RMBS
Financial Assets:
Available-for-sale securities	0	0
Fair value measurements on recurring basis | Level I | CMBS
Financial Assets:
Available-for-sale securities	0	0
Fair value measurements on recurring basis | Level I | Domestic servicing rights, at fair value
Financial Assets:
Domestic servicing rights	0	0
Fair value measurements on recurring basis | Level II
Financial Assets:
Equity security, fair value	0	0
Derivative assets	35,615	40,555
Total Assets	286,302	40,555
Liabilities, Fair Value Disclosure [Abstract]
Derivative liabilities	33,061	41,324
Total Liabilities	57,198,579	60,797,819
Fair value measurements on recurring basis | Level II | Loans held-for-sale
Financial Assets:
Loans under fair value option	250,687	0
Fair value measurements on recurring basis | Level II | RMBS
Financial Assets:
Available-for-sale securities	0	0
Fair value measurements on recurring basis | Level II | CMBS
Financial Assets:
Available-for-sale securities	0	0
Fair value measurements on recurring basis | Level II | Domestic servicing rights, at fair value
Financial Assets:
Domestic servicing rights	0	0
Fair value measurements on recurring basis | Level III
Financial Assets:
Equity security, fair value	0	0
Derivative assets	0	0
Total Assets	64,448,991	65,461,315
Liabilities, Fair Value Disclosure [Abstract]
Derivative liabilities	0	0
Total Liabilities	4,836,192	2,019,876
Fair value measurements on recurring basis | Level III | Loans held-for-sale
Financial Assets:
Loans under fair value option	767,821	1,022,979
Fair value measurements on recurring basis | Level III | RMBS
Financial Assets:
Available-for-sale securities	154,704	167,349
Fair value measurements on recurring basis | Level III | CMBS
Financial Assets:
Available-for-sale securities	18,965	19,457
Fair value measurements on recurring basis | Level III | Domestic servicing rights, at fair value
Financial Assets:
Domestic servicing rights	13,705	13,202
Primary beneficiary
Financial Assets:
Assets	63,493,796	64,238,328
Liabilities, Fair Value Disclosure [Abstract]
Liabilities	62,001,710	62,776,371
Primary beneficiary | Fair value measurements on recurring basis | VIE assets
Financial Assets:
Assets	63,493,796	64,238,328
Primary beneficiary | Fair value measurements on recurring basis | VIE liabilities
Liabilities, Fair Value Disclosure [Abstract]
Liabilities	62,001,710	62,776,371
Primary beneficiary | Fair value measurements on recurring basis | Level I | VIE assets
Financial Assets:
Assets	0	0
Primary beneficiary | Fair value measurements on recurring basis | Level I | VIE liabilities
Liabilities, Fair Value Disclosure [Abstract]
Liabilities	0	0
Primary beneficiary | Fair value measurements on recurring basis | Level II | VIE assets
Financial Assets:
Assets	0	0
Primary beneficiary | Fair value measurements on recurring basis | Level II | VIE liabilities
Liabilities, Fair Value Disclosure [Abstract]
Liabilities	57,165,518	60,756,495
Primary beneficiary | Fair value measurements on recurring basis | Level III | VIE assets
Financial Assets:
Assets	63,493,796	64,238,328
Primary beneficiary | Fair value measurements on recurring basis | Level III | VIE liabilities
Liabilities, Fair Value Disclosure [Abstract]
Liabilities	$ 4,836,192	$ 2,019,876