Fair Value - Significant Unobservable Inputs (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2023 USD ($)	Dec. 31, 2022 USD ($)
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities	$ 102,368	$ 113,386
Woodstar Fund investments		1,761,002
Domestic servicing rights	19,384	17,790
VIE assets	69,504,196	79,043,129
VIE liabilities	62,481,214	71,844,422
Primary beneficiary
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Woodstar Fund investments	2,012,833	1,761,002
VIE assets	47,800
VIE liabilities	10,300
Fair value measurements on recurring basis | Loans under fair value option
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans under fair value option	2,645,637	2,784,594
Fair value measurements on recurring basis | RMBS
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities	102,368	113,386
Fair value measurements on recurring basis | CMBS
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities	18,600	19,108
Fair value measurements on recurring basis | Woodstar Fund investments
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Woodstar Fund investments	2,012,833	1,761,002
Fair value measurements on recurring basis | Domestic servicing rights, at fair value
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Domestic servicing rights	19,384	17,790
Fair value measurements on recurring basis | VIE assets | Primary beneficiary
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE assets	43,786,356	52,453,041
Fair value measurements on recurring basis | VIE liabilities | Primary beneficiary
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE liabilities	42,175,734	50,754,355
Fair value measurements on recurring basis | Level III | Loans under fair value option
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans under fair value option	$ 2,645,637	$ 2,784,594
Fair value measurements on recurring basis | Level III | Loans under fair value option | Minimum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans under fair value option, duration	4 years 3 months 18 days	5 years 3 months 18 days
Fair value measurements on recurring basis | Level III | Loans under fair value option | Minimum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans under fair value option	0.028	0.028
Fair value measurements on recurring basis | Level III | Loans under fair value option | Minimum | FICO score
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans under fair value option	585	585
Fair value measurements on recurring basis | Level III | Loans under fair value option | Minimum | LTV
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans under fair value option	0.05	0.04
Fair value measurements on recurring basis | Level III | Loans under fair value option | Minimum | Purchase price
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans under fair value option	0.800	0.800
Fair value measurements on recurring basis | Level III | Loans under fair value option | Maximum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans under fair value option, duration	38 years 6 months	39 years 6 months
Fair value measurements on recurring basis | Level III | Loans under fair value option | Maximum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans under fair value option	0.099	0.093
Fair value measurements on recurring basis | Level III | Loans under fair value option | Maximum | FICO score
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans under fair value option	900	900
Fair value measurements on recurring basis | Level III | Loans under fair value option | Maximum | LTV
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans under fair value option	1.40	0.92
Fair value measurements on recurring basis | Level III | Loans under fair value option | Maximum | Purchase price
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans under fair value option	1.086	1.086
Fair value measurements on recurring basis | Level III | Loans under fair value option | Weighted-average
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans under fair value option, duration	27 years 4 months 24 days	28 years 7 months 6 days
Fair value measurements on recurring basis | Level III | Loans under fair value option | Weighted-average | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans under fair value option	0.045	0.045
Fair value measurements on recurring basis | Level III | Loans under fair value option | Weighted-average | FICO score
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans under fair value option	749	749
Fair value measurements on recurring basis | Level III | Loans under fair value option | Weighted-average | LTV
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans under fair value option	0.68	0.67
Fair value measurements on recurring basis | Level III | Loans under fair value option | Weighted-average | Purchase price
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans under fair value option	1.014	1.014
Fair value measurements on recurring basis | Level III | RMBS
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities	$ 102,368	$ 113,386
Loss severity for specified percentage of portfolio (as a percent)	5.00%	10.00%
Fair value measurements on recurring basis | Level III | RMBS | Minimum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loss severity for specified percentage of portfolio (as a percent)	45.00%	45.00%
Fair value measurements on recurring basis | Level III | RMBS | Minimum | Constant prepayment rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.029	0.028
Fair value measurements on recurring basis | Level III | RMBS | Minimum | Constant default rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.010	0.011
Fair value measurements on recurring basis | Level III | RMBS | Minimum | Loss severity
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0	0
Fair value measurements on recurring basis | Level III | RMBS | Minimum | Delinquency rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.08	0.06
Fair value measurements on recurring basis | Level III | RMBS | Minimum | Servicer advances
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.30	0.31
Fair value measurements on recurring basis | Level III | RMBS | Minimum | Annual coupon deterioration
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0	0
Fair value measurements on recurring basis | Level III | RMBS | Minimum | Putback amount per projected total collateral loss
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0	0
Fair value measurements on recurring basis | Level III | RMBS | Maximum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loss severity for specified percentage of portfolio (as a percent)	80.00%	80.00%
Fair value measurements on recurring basis | Level III | RMBS | Maximum | Constant prepayment rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.096	0.120
Fair value measurements on recurring basis | Level III | RMBS | Maximum | Constant default rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.042	0.044
Fair value measurements on recurring basis | Level III | RMBS | Maximum | Loss severity
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.99	1.09
Fair value measurements on recurring basis | Level III | RMBS | Maximum | Delinquency rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.25	0.29
Fair value measurements on recurring basis | Level III | RMBS | Maximum | Servicer advances
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.78	0.777
Fair value measurements on recurring basis | Level III | RMBS | Maximum | Annual coupon deterioration
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.013	0.026
Fair value measurements on recurring basis | Level III | RMBS | Maximum | Putback amount per projected total collateral loss
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.08	0.08
Fair value measurements on recurring basis | Level III | RMBS | Weighted-average | Constant prepayment rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.052	0.055
Fair value measurements on recurring basis | Level III | RMBS | Weighted-average | Constant default rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.017	0.020
Fair value measurements on recurring basis | Level III | RMBS | Weighted-average | Loss severity
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.17	0.24
Fair value measurements on recurring basis | Level III | RMBS | Weighted-average | Delinquency rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.14	0.16
Fair value measurements on recurring basis | Level III | RMBS | Weighted-average | Servicer advances
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.51	0.53
Fair value measurements on recurring basis | Level III | RMBS | Weighted-average | Annual coupon deterioration
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.001	0.001
Fair value measurements on recurring basis | Level III | RMBS | Weighted-average | Putback amount per projected total collateral loss
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.005	0.005
Fair value measurements on recurring basis | Level III | CMBS
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities	$ 18,600	$ 19,108
Fair value measurements on recurring basis | Level III | CMBS | Minimum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
CMBS, term	0 years	0 years
Fair value measurements on recurring basis | Level III | CMBS | Minimum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0	0
Fair value measurements on recurring basis | Level III | CMBS | Maximum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
CMBS, term	6 years 8 months 12 days	7 years 8 months 12 days
Fair value measurements on recurring basis | Level III | CMBS | Maximum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	5.401	1.175
Fair value measurements on recurring basis | Level III | CMBS | Weighted-average
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
CMBS, term	2 years 4 months 24 days	3 years
Fair value measurements on recurring basis | Level III | CMBS | Weighted-average | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.106	0.101
Fair value measurements on recurring basis | Level III | Woodstar Fund investments
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Woodstar Fund investments	$ 2,012,833	$ 1,761,002
Fair value measurements on recurring basis | Level III | Woodstar Fund investments | Implied direct capitalization rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.0425	0.0420
Fair value measurements on recurring basis | Level III | Woodstar Fund investments | Minimum | Discount rate - properties
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.063	0.063
Fair value measurements on recurring basis | Level III | Woodstar Fund investments | Minimum | Discount rate - debt
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.030	0.056
Fair value measurements on recurring basis | Level III | Woodstar Fund investments | Minimum | Terminal capitalization rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.048	0.050
Fair value measurements on recurring basis | Level III | Woodstar Fund investments | Maximum | Discount rate - properties
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.070	0.068
Fair value measurements on recurring basis | Level III | Woodstar Fund investments | Maximum | Discount rate - debt
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.069	0.067
Fair value measurements on recurring basis | Level III | Woodstar Fund investments | Maximum | Terminal capitalization rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.055	0.055
Fair value measurements on recurring basis | Level III | Woodstar Fund investments | Weighted-average | Discount rate - properties
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.067	0.065
Fair value measurements on recurring basis | Level III | Woodstar Fund investments | Weighted-average | Discount rate - debt
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.054	0.061
Fair value measurements on recurring basis | Level III | Woodstar Fund investments | Weighted-average | Terminal capitalization rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.052	0.051
Fair value measurements on recurring basis | Level III | Woodstar Fund investments | Weighted-average | Implied direct capitalization rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.0425	0.0420
Fair value measurements on recurring basis | Level III | Domestic servicing rights, at fair value
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Domestic servicing rights	$ 19,384	$ 17,790
Fair value measurements on recurring basis | Level III | Domestic servicing rights, at fair value | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Domestic servicing rights	0.0850	0.0825
Fair value measurements on recurring basis | Level III | Domestic servicing rights, at fair value | Discount rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Domestic servicing rights	0.15	0.15
Fair value measurements on recurring basis | Level III | Domestic servicing rights, at fair value | Weighted-average | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Domestic servicing rights	0.0850	0.0825
Fair value measurements on recurring basis | Level III | Domestic servicing rights, at fair value | Weighted-average | Discount rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Domestic servicing rights	0.15	0.15
Fair value measurements on recurring basis | Level III | VIE assets | Minimum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE duration (in years)	0 years	0 years
Fair value measurements on recurring basis | Level III | VIE assets | Minimum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE assets	0	0
Fair value measurements on recurring basis | Level III | VIE assets | Maximum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE duration (in years)	10 years	11 years
Fair value measurements on recurring basis | Level III | VIE assets | Maximum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE assets	6.910	4.536
Fair value measurements on recurring basis | Level III | VIE assets | Weighted-average
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE duration (in years)	1 year 9 months 18 days	2 years 4 months 24 days
Fair value measurements on recurring basis | Level III | VIE assets | Weighted-average | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE assets	0.159	0.153
Fair value measurements on recurring basis | Level III | VIE assets | Primary beneficiary
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE assets	$ 43,786,356	$ 52,453,041
Fair value measurements on recurring basis | Level III | VIE liabilities | Minimum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE duration (in years)	0 years	0 years
Fair value measurements on recurring basis | Level III | VIE liabilities | Minimum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE liabilities	0	0
Fair value measurements on recurring basis | Level III | VIE liabilities | Maximum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE duration (in years)	10 years	11 years
Fair value measurements on recurring basis | Level III | VIE liabilities | Maximum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE liabilities	6.910	4.536
Fair value measurements on recurring basis | Level III | VIE liabilities | Weighted-average
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE duration (in years)	1 year 8 months 12 days	1 year 9 months 18 days
Fair value measurements on recurring basis | Level III | VIE liabilities | Weighted-average | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE liabilities	0.114	0.104
Fair value measurements on recurring basis | Level III | VIE liabilities | Primary beneficiary
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE liabilities	$ 5,604,796	$ 5,505,943