Fair Value - Significant Unobservable Inputs (Details) $ in Thousands	3 Months Ended	12 Months Ended
	Mar. 31, 2022 USD ($) score	Dec. 31, 2021 USD ($) score
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities	$ 134,406	$ 143,980
Domestic servicing rights	17,864	16,780
Assets	81,187,898	83,850,397
Liabilities	74,278,245	77,201,590
Fair value measurements on recurring basis | Loans held-for-sale
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans under fair value option	2,740,133	2,936,025
Fair value measurements on recurring basis | RMBS
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities	134,406	143,980
Fair value measurements on recurring basis | CMBS
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities	21,858	22,244
Fair value measurements on recurring basis | Woodstar Fund investments
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Investments	1,264,160	1,040,309
Fair value measurements on recurring basis | Domestic servicing rights, at fair value
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Domestic servicing rights	17,864	16,780
Fair value measurements on recurring basis | Level III | Loans held-for-sale
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans under fair value option	$ 2,367,022	$ 2,936,025
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Minimum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, duration	6 years	6 years 3 months 18 days
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Minimum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, measurement input	0.028	0.026
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Minimum | FICO score
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, measurement input | score	584	582
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Minimum | LTV
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, measurement input	0.04	0.01
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Minimum | Purchase price
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, measurement input	0.800	0.800
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Maximum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, duration	39 years 10 months 24 days	39 years 10 months 24 days
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Maximum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, measurement input	0.092	0.092
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Maximum | FICO score
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, measurement input | score	829	829
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Maximum | LTV
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, measurement input	0.94	0.94
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Maximum | Purchase price
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, measurement input	1.086	1.086
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Weighted-average
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, duration	28 years 10 months 24 days	27 years 4 months 24 days
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Weighted-average | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, measurement input	0.044	0.042
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Weighted-average | FICO score
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, measurement input | score	745	748
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Weighted-average | LTV
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, measurement input	0.68	0.66
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Weighted-average | Purchase price
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, measurement input	1.020	1.023
Fair value measurements on recurring basis | Level III | RMBS
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities	$ 134,406	$ 143,980
Loss severity for specified percentage of portfolio (as a percent)	11.00%	18.00%
Fair value measurements on recurring basis | Level III | RMBS | Minimum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loss severity for specified percentage of portfolio (as a percent)	45.00%
Fair value measurements on recurring basis | Level III | RMBS | Minimum | Constant prepayment rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.042	0.048
Fair value measurements on recurring basis | Level III | RMBS | Minimum | Constant default rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.008	0.008
Fair value measurements on recurring basis | Level III | RMBS | Minimum | Loss severity
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0	0
Fair value measurements on recurring basis | Level III | RMBS | Minimum | Delinquency rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.11	0.104
Fair value measurements on recurring basis | Level III | RMBS | Minimum | Servicer advances
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.22	0.19
Fair value measurements on recurring basis | Level III | RMBS | Minimum | Annual coupon deterioration
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0	0
Fair value measurements on recurring basis | Level III | RMBS | Minimum | Putback amount per projected total collateral loss
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0	0
Fair value measurements on recurring basis | Level III | RMBS | Maximum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loss severity for specified percentage of portfolio (as a percent)	80.00%
Fair value measurements on recurring basis | Level III | RMBS | Maximum | Constant prepayment rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.196	0.192
Fair value measurements on recurring basis | Level III | RMBS | Maximum | Constant default rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.063	0.060
Fair value measurements on recurring basis | Level III | RMBS | Maximum | Loss severity
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.81	0.86
Fair value measurements on recurring basis | Level III | RMBS | Maximum | Delinquency rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.36	0.35
Fair value measurements on recurring basis | Level III | RMBS | Maximum | Servicer advances
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.79	0.83
Fair value measurements on recurring basis | Level III | RMBS | Maximum | Annual coupon deterioration
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.025	0.017
Fair value measurements on recurring basis | Level III | RMBS | Maximum | Putback amount per projected total collateral loss
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.08	0.08
Fair value measurements on recurring basis | Level III | RMBS | Weighted-average | Constant prepayment rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.093	0.099
Fair value measurements on recurring basis | Level III | RMBS | Weighted-average | Constant default rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.020	0.021
Fair value measurements on recurring basis | Level III | RMBS | Weighted-average | Loss severity
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.24	0.26
Fair value measurements on recurring basis | Level III | RMBS | Weighted-average | Delinquency rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.18	0.19
Fair value measurements on recurring basis | Level III | RMBS | Weighted-average | Servicer advances
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.52	0.52
Fair value measurements on recurring basis | Level III | RMBS | Weighted-average | Annual coupon deterioration
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.001	0.001
Fair value measurements on recurring basis | Level III | RMBS | Weighted-average | Putback amount per projected total collateral loss
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.005	0.005
Fair value measurements on recurring basis | Level III | CMBS
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities	$ 21,858	$ 22,244
Fair value measurements on recurring basis | Level III | CMBS | Minimum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale debt securities, term	0 years	0 days
Fair value measurements on recurring basis | Level III | CMBS | Minimum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0	0
Fair value measurements on recurring basis | Level III | CMBS | Maximum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale debt securities, term	6 years 8 months 12 days	7 years 2 months 12 days
Fair value measurements on recurring basis | Level III | CMBS | Maximum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	2.978	6.136
Fair value measurements on recurring basis | Level III | CMBS | Weighted-average
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale debt securities, term	3 years 8 months 12 days	5 years 2 months 12 days
Fair value measurements on recurring basis | Level III | CMBS | Weighted-average | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.091	0.093
Fair value measurements on recurring basis | Level III | Woodstar Fund investments
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Investments	$ 1,264,160	$ 1,040,309
Fair value measurements on recurring basis | Level III | Woodstar Fund investments | Minimum | Discount rate - properties
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input		0.058
Fair value measurements on recurring basis | Level III | Woodstar Fund investments | Minimum | Discount rate - debt
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.023	0.026
Fair value measurements on recurring basis | Level III | Woodstar Fund investments | Minimum | Terminal capitalization rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input		0.048
Fair value measurements on recurring basis | Level III | Woodstar Fund investments | Minimum | Direct capitalization rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.041	0.041
Fair value measurements on recurring basis | Level III | Woodstar Fund investments | Maximum | Discount rate - properties
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input		0.063
Fair value measurements on recurring basis | Level III | Woodstar Fund investments | Maximum | Discount rate - debt
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.044	0.033
Fair value measurements on recurring basis | Level III | Woodstar Fund investments | Maximum | Terminal capitalization rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input		0.053
Fair value measurements on recurring basis | Level III | Woodstar Fund investments | Maximum | Direct capitalization rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.042	0.042
Fair value measurements on recurring basis | Level III | Woodstar Fund investments | Weighted-average | Discount rate - properties
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input		0.060
Fair value measurements on recurring basis | Level III | Woodstar Fund investments | Weighted-average | Discount rate - debt
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.038	0.029
Fair value measurements on recurring basis | Level III | Woodstar Fund investments | Weighted-average | Terminal capitalization rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input		0.049
Fair value measurements on recurring basis | Level III | Woodstar Fund investments | Weighted-average | Direct capitalization rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.042	0.042
Fair value measurements on recurring basis | Level III | Domestic servicing rights, at fair value
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Domestic servicing rights	$ 17,864	$ 16,780
Fair value measurements on recurring basis | Level III | Domestic servicing rights, at fair value | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Servicing asset, measurement input	0.0750	0.0730
Fair value measurements on recurring basis | Level III | Domestic servicing rights, at fair value | Discount rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Servicing asset, measurement input	0.15	0.15
Fair value measurements on recurring basis | Level III | Domestic servicing rights, at fair value | Weighted-average | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Servicing asset, measurement input	0.0750	0.0730
Fair value measurements on recurring basis | Level III | Domestic servicing rights, at fair value | Weighted-average | Discount rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Servicing asset, measurement input	0.15	0.15
Fair value measurements on recurring basis | Level III | VIE assets | Minimum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE duration (in years)	0 years	0 days
Fair value measurements on recurring basis | Level III | VIE assets | Minimum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE assets, measurement input	0	0
Fair value measurements on recurring basis | Level III | VIE assets | Maximum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE duration (in years)	10 years 10 months 24 days	11 years
Fair value measurements on recurring basis | Level III | VIE assets | Maximum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE assets, measurement input	3.286	6.153
Fair value measurements on recurring basis | Level III | VIE assets | Weighted-average
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE duration (in years)	3 years 1 month 6 days	3 years 2 months 12 days
Fair value measurements on recurring basis | Level III | VIE assets | Weighted-average | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE assets, measurement input	0.150	0.130
Fair value measurements on recurring basis | Level III | VIE liabilities | Minimum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE duration (in years)	0 years	0 days
Fair value measurements on recurring basis | Level III | VIE liabilities | Minimum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE liabilities, measurement input	0	0
Fair value measurements on recurring basis | Level III | VIE liabilities | Maximum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE duration (in years)	10 years 10 months 24 days	11 years
Fair value measurements on recurring basis | Level III | VIE liabilities | Maximum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE liabilities, measurement input	3.286	6.153
Fair value measurements on recurring basis | Level III | VIE liabilities | Weighted-average
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE duration (in years)	2 years 2 months 12 days	2 years 3 months 18 days
Fair value measurements on recurring basis | Level III | VIE liabilities | Weighted-average | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE liabilities, measurement input	0.076	0.064
Primary beneficiary
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Investments	$ 1,264,160	$ 1,040,309
Assets	75,400
Liabilities	30,800
Primary beneficiary | Fair value measurements on recurring basis | VIE assets
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Assets	57,763,543	61,280,543
Primary beneficiary | Fair value measurements on recurring basis | VIE liabilities
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Liabilities	56,185,937	59,752,922
Primary beneficiary | Fair value measurements on recurring basis | Level III | VIE assets
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Assets	57,763,543	61,280,543
Primary beneficiary | Fair value measurements on recurring basis | Level III | VIE liabilities
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Liabilities	$ 5,479,288	$ 4,780,221