Fair Value - Financial Assets and Liabilities (Details) - USD ($) $ in Thousands	Mar. 31, 2022	Dec. 31, 2021
Financial Assets:
Available-for-sale securities	$ 134,406	$ 143,980
Domestic servicing rights	17,864	16,780
Derivative assets	60,869	48,216
Assets	81,187,898	83,850,397
Liabilities, Fair Value Disclosure [Abstract]
Derivative liabilities	38,709	13,421
Liabilities	74,278,245	77,201,590
Fair value measurements on recurring basis
Financial Assets:
Equity security, fair value	11,619	11,624
Derivative assets	60,869	48,216
Total Assets	62,014,452	65,499,721
Liabilities, Fair Value Disclosure [Abstract]
Derivative liabilities	38,709	13,421
Total Liabilities	56,224,646	59,766,343
Fair value measurements on recurring basis | Loans held-for-sale
Financial Assets:
Loans under fair value option	2,740,133	2,936,025
Fair value measurements on recurring basis | RMBS
Financial Assets:
Available-for-sale securities	134,406	143,980
Fair value measurements on recurring basis | CMBS
Financial Assets:
Available-for-sale securities	21,858	22,244
Fair value measurements on recurring basis | Woodstar Fund investments
Financial Assets:
Investments	1,264,160	1,040,309
Fair value measurements on recurring basis | Domestic servicing rights, at fair value
Financial Assets:
Domestic servicing rights	17,864	16,780
Fair value measurements on recurring basis | Level I
Financial Assets:
Equity security, fair value	11,619	11,624
Derivative assets	0	0
Total Assets	11,619	11,624
Liabilities, Fair Value Disclosure [Abstract]
Derivative liabilities	0	0
Total Liabilities	0	0
Fair value measurements on recurring basis | Level I | Loans held-for-sale
Financial Assets:
Loans under fair value option	0	0
Fair value measurements on recurring basis | Level I | RMBS
Financial Assets:
Available-for-sale securities	0	0
Fair value measurements on recurring basis | Level I | CMBS
Financial Assets:
Available-for-sale securities	0	0
Fair value measurements on recurring basis | Level I | Woodstar Fund investments
Financial Assets:
Investments	0	0
Fair value measurements on recurring basis | Level I | Domestic servicing rights, at fair value
Financial Assets:
Domestic servicing rights	0	0
Fair value measurements on recurring basis | Level II
Financial Assets:
Equity security, fair value	0	0
Derivative assets	60,869	48,216
Total Assets	433,980	48,216
Liabilities, Fair Value Disclosure [Abstract]
Derivative liabilities	38,709	13,421
Total Liabilities	50,745,358	54,986,122
Fair value measurements on recurring basis | Level II | Loans held-for-sale
Financial Assets:
Loans under fair value option	373,111	0
Fair value measurements on recurring basis | Level II | RMBS
Financial Assets:
Available-for-sale securities	0	0
Fair value measurements on recurring basis | Level II | CMBS
Financial Assets:
Available-for-sale securities	0	0
Fair value measurements on recurring basis | Level II | Woodstar Fund investments
Financial Assets:
Investments	0	0
Fair value measurements on recurring basis | Level II | Domestic servicing rights, at fair value
Financial Assets:
Domestic servicing rights	0	0
Fair value measurements on recurring basis | Level III
Financial Assets:
Equity security, fair value	0	0
Derivative assets	0	0
Total Assets	61,568,853	65,439,881
Liabilities, Fair Value Disclosure [Abstract]
Derivative liabilities	0	0
Total Liabilities	5,479,288	4,780,221
Fair value measurements on recurring basis | Level III | Loans held-for-sale
Financial Assets:
Loans under fair value option	2,367,022	2,936,025
Fair value measurements on recurring basis | Level III | RMBS
Financial Assets:
Available-for-sale securities	134,406	143,980
Fair value measurements on recurring basis | Level III | CMBS
Financial Assets:
Available-for-sale securities	21,858	22,244
Fair value measurements on recurring basis | Level III | Woodstar Fund investments
Financial Assets:
Investments	1,264,160	1,040,309
Fair value measurements on recurring basis | Level III | Domestic servicing rights, at fair value
Financial Assets:
Domestic servicing rights	17,864	16,780
Primary beneficiary
Financial Assets:
Investments	1,264,160	1,040,309
Assets	75,400
Liabilities, Fair Value Disclosure [Abstract]
Liabilities	30,800
Primary beneficiary | Fair value measurements on recurring basis | VIE assets
Financial Assets:
Assets	57,763,543	61,280,543
Primary beneficiary | Fair value measurements on recurring basis | VIE liabilities
Liabilities, Fair Value Disclosure [Abstract]
Liabilities	56,185,937	59,752,922
Primary beneficiary | Fair value measurements on recurring basis | Level I | VIE assets
Financial Assets:
Assets	0	0
Primary beneficiary | Fair value measurements on recurring basis | Level I | VIE liabilities
Liabilities, Fair Value Disclosure [Abstract]
Liabilities	0	0
Primary beneficiary | Fair value measurements on recurring basis | Level II | VIE assets
Financial Assets:
Assets	0	0
Primary beneficiary | Fair value measurements on recurring basis | Level II | VIE liabilities
Liabilities, Fair Value Disclosure [Abstract]
Liabilities	50,706,649	54,972,701
Primary beneficiary | Fair value measurements on recurring basis | Level III | VIE assets
Financial Assets:
Assets	57,763,543	61,280,543
Primary beneficiary | Fair value measurements on recurring basis | Level III | VIE liabilities
Liabilities, Fair Value Disclosure [Abstract]
Liabilities	$ 5,479,288	$ 4,780,221