Fair Value - Financial Assets and Liabilities (Details) - USD ($) $ in Thousands	Dec. 31, 2021	Dec. 31, 2020
Financial Assets:
Available-for-sale securities	$ 143,980	$ 167,349
Domestic servicing rights	16,780	13,202
Derivative assets	48,216	40,555
Assets	83,850,397	80,873,509
Liabilities, Fair Value Disclosure [Abstract]
Derivative liabilities	13,421	41,324
Liabilities	77,201,590	76,010,933
Fair value measurements on recurring basis
Financial Assets:
Equity security, fair value	11,624	11,247
Derivative assets	48,216	40,555
Total Assets	65,499,721	65,513,117
Liabilities, Fair Value Disclosure [Abstract]
Derivative liabilities	13,421	41,324
Total Liabilities	59,766,343	62,817,695
Fair value measurements on recurring basis | Loans held-for-sale
Financial Assets:
Loans under fair value option	2,936,025	1,022,979
Fair value measurements on recurring basis | RMBS
Financial Assets:
Available-for-sale securities	143,980	167,349
Fair value measurements on recurring basis | CMBS
Financial Assets:
Available-for-sale securities	22,244	19,457
Fair value measurements on recurring basis | Woodstar Fund investments
Financial Assets:
Investments	1,040,309
Fair value measurements on recurring basis | Domestic servicing rights, at fair value
Financial Assets:
Domestic servicing rights	16,780	13,202
Fair value measurements on recurring basis | Level I
Financial Assets:
Equity security, fair value	11,624	11,247
Derivative assets	0	0
Total Assets	11,624	11,247
Liabilities, Fair Value Disclosure [Abstract]
Derivative liabilities	0	0
Total Liabilities	0	0
Fair value measurements on recurring basis | Level I | Loans held-for-sale
Financial Assets:
Loans under fair value option	0	0
Fair value measurements on recurring basis | Level I | RMBS
Financial Assets:
Available-for-sale securities	0	0
Fair value measurements on recurring basis | Level I | CMBS
Financial Assets:
Available-for-sale securities	0	0
Fair value measurements on recurring basis | Level I | Woodstar Fund investments
Financial Assets:
Investments	0
Fair value measurements on recurring basis | Level I | Domestic servicing rights, at fair value
Financial Assets:
Domestic servicing rights	0	0
Fair value measurements on recurring basis | Level II
Financial Assets:
Equity security, fair value	0	0
Derivative assets	48,216	40,555
Total Assets	48,216	40,555
Liabilities, Fair Value Disclosure [Abstract]
Derivative liabilities	13,421	41,324
Total Liabilities	54,986,122	60,797,819
Fair value measurements on recurring basis | Level II | Loans held-for-sale
Financial Assets:
Loans under fair value option	0	0
Fair value measurements on recurring basis | Level II | RMBS
Financial Assets:
Available-for-sale securities	0	0
Fair value measurements on recurring basis | Level II | CMBS
Financial Assets:
Available-for-sale securities	0	0
Fair value measurements on recurring basis | Level II | Woodstar Fund investments
Financial Assets:
Investments	0
Fair value measurements on recurring basis | Level II | Domestic servicing rights, at fair value
Financial Assets:
Domestic servicing rights	0	0
Fair value measurements on recurring basis | Level III
Financial Assets:
Equity security, fair value	0	0
Derivative assets	0	0
Total Assets	65,439,881	65,461,315
Liabilities, Fair Value Disclosure [Abstract]
Derivative liabilities	0	0
Total Liabilities	4,780,221	2,019,876
Fair value measurements on recurring basis | Level III | Loans held-for-sale
Financial Assets:
Loans under fair value option	2,936,025	1,022,979
Fair value measurements on recurring basis | Level III | RMBS
Financial Assets:
Available-for-sale securities	143,980	167,349
Fair value measurements on recurring basis | Level III | CMBS
Financial Assets:
Available-for-sale securities	22,244	19,457
Fair value measurements on recurring basis | Level III | Woodstar Fund investments
Financial Assets:
Investments	1,040,309
Fair value measurements on recurring basis | Level III | Domestic servicing rights, at fair value
Financial Assets:
Domestic servicing rights	16,780	13,202
Primary beneficiary
Financial Assets:
Investments	1,040,309	0
Assets	102,400
Liabilities, Fair Value Disclosure [Abstract]
Liabilities	53,500
Primary beneficiary | Fair value measurements on recurring basis | VIE assets
Financial Assets:
Assets	61,280,543	64,238,328
Primary beneficiary | Fair value measurements on recurring basis | VIE liabilities
Liabilities, Fair Value Disclosure [Abstract]
Liabilities	59,752,922	62,776,371
Primary beneficiary | Fair value measurements on recurring basis | Level I | VIE assets
Financial Assets:
Assets	0	0
Primary beneficiary | Fair value measurements on recurring basis | Level I | VIE liabilities
Liabilities, Fair Value Disclosure [Abstract]
Liabilities	0	0
Primary beneficiary | Fair value measurements on recurring basis | Level II | VIE assets
Financial Assets:
Assets	0	0
Primary beneficiary | Fair value measurements on recurring basis | Level II | VIE liabilities
Liabilities, Fair Value Disclosure [Abstract]
Liabilities	54,972,701	60,756,495
Primary beneficiary | Fair value measurements on recurring basis | Level III | VIE assets
Financial Assets:
Assets	61,280,543	64,238,328
Primary beneficiary | Fair value measurements on recurring basis | Level III | VIE liabilities
Liabilities, Fair Value Disclosure [Abstract]
Liabilities	$ 4,780,221	$ 2,019,876