Fair Value - Significant unobservable inputs (Details)	3 Months Ended	12 Months Ended
	Mar. 31, 2021 USD ($) item	Dec. 31, 2020 USD ($) item
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Domestic servicing rights	$ 12,406,000	$ 13,202,000
Total Assets	79,702,323,000	80,873,509,000
Total Liabilities	74,847,538,000	76,010,933,000
Fair value measurements on recurring basis | Loans held-for-sale
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, fair value option	763,773,000	1,022,979,000
Fair value measurements on recurring basis | RMBS
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities	160,301,000	167,349,000
Fair value measurements on recurring basis | CMBS
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities	19,256,000	19,457,000
Fair value measurements on recurring basis | Domestic Servicing Rights
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Domestic servicing rights	12,406,000	13,202,000
Fair value measurements on recurring basis | Level III | Loans held-for-sale
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, fair value option	$ 763,773,000	$ 1,022,979,000
Loans Held-for-sale, Valuation Technique [Extensible List]	us-gaap:ValuationTechniqueDiscountedCashFlowMember	us-gaap:ValuationTechniqueDiscountedCashFlowMember
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Minimum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, duration	7 years	7 years 3 months 18 days
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Minimum | FICO score
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, measurement input | item	519	519
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Minimum | LTV
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, measurement input	15	5
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Minimum | Purchase price
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, measurement input	85.6	84.4
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Minimum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, measurement input	3.4	3.3
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Maximum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, duration	39 years	39 years 3 months 18 days
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Maximum | FICO score
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, measurement input | item	823	823
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Maximum | LTV
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, measurement input	94	94
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Maximum | Purchase price
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, measurement input	104.8	104.8
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Maximum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, measurement input	9.5	9.7
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Weighted-average
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, duration	24 years 8 months 12 days	26 years 3 months 18 days
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Weighted-average | FICO score
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, measurement input	(732)	(727)
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Weighted-average | LTV
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, measurement input	66	68
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Weighted-average | Purchase price
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, measurement input	101.7	99.8
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Weighted-average | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, measurement input	5.5	5.9
Fair value measurements on recurring basis | Level III | RMBS
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities	$ 160,301,000	$ 167,349,000
Debt Securities, Available-for-sale, Valuation Technique [Extensible List]	us-gaap:ValuationTechniqueDiscountedCashFlowMember	us-gaap:ValuationTechniqueDiscountedCashFlowMember
Loss severity for specified percentage of portfolio (as a percent)	14.00%	23.00%
Fair value measurements on recurring basis | Level III | RMBS | Minimum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loss severity for specified percentage of portfolio (as a percent)	45.00%	45.00%
Fair value measurements on recurring basis | Level III | RMBS | Minimum | Constant prepayment rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	3.5	3.6
Fair value measurements on recurring basis | Level III | RMBS | Minimum | Constant default rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.7	0.7
Fair value measurements on recurring basis | Level III | RMBS | Minimum | Loss severity
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0	0
Fair value measurements on recurring basis | Level III | RMBS | Minimum | Delinquency rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	9	10
Fair value measurements on recurring basis | Level III | RMBS | Minimum | Servicer advances
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	23	23
Fair value measurements on recurring basis | Level III | RMBS | Minimum | Annual coupon deterioration
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.0	0.0
Fair value measurements on recurring basis | Level III | RMBS | Minimum | Putback amount per projected total collateral loss
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0	0
Fair value measurements on recurring basis | Level III | RMBS | Maximum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loss severity for specified percentage of portfolio (as a percent)	80.00%	80.00%
Fair value measurements on recurring basis | Level III | RMBS | Maximum | Constant prepayment rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	17.3	19.4
Fair value measurements on recurring basis | Level III | RMBS | Maximum | Constant default rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	5.0	5.4
Fair value measurements on recurring basis | Level III | RMBS | Maximum | Loss severity
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	84	85
Fair value measurements on recurring basis | Level III | RMBS | Maximum | Delinquency rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	32	32
Fair value measurements on recurring basis | Level III | RMBS | Maximum | Servicer advances
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	84	82
Fair value measurements on recurring basis | Level III | RMBS | Maximum | Annual coupon deterioration
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	1.2	0.9
Fair value measurements on recurring basis | Level III | RMBS | Maximum | Putback amount per projected total collateral loss
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	17	17
Fair value measurements on recurring basis | Level III | RMBS | Weighted-average | Constant prepayment rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	7.4	7.6
Fair value measurements on recurring basis | Level III | RMBS | Weighted-average | Constant default rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	2.2	2.4
Fair value measurements on recurring basis | Level III | RMBS | Weighted-average | Loss severity
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	17	20
Fair value measurements on recurring basis | Level III | RMBS | Weighted-average | Delinquency rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	18	19
Fair value measurements on recurring basis | Level III | RMBS | Weighted-average | Servicer advances
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	53	54
Fair value measurements on recurring basis | Level III | RMBS | Weighted-average | Annual coupon deterioration
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.1	0.1
Fair value measurements on recurring basis | Level III | RMBS | Weighted-average | Putback amount per projected total collateral loss
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.8	0.8
Fair value measurements on recurring basis | Level III | CMBS
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities	$ 19,256,000	$ 19,457,000
Debt Securities, Available-for-sale, Valuation Technique [Extensible List]	us-gaap:ValuationTechniqueDiscountedCashFlowMember	us-gaap:ValuationTechniqueDiscountedCashFlowMember
Fair value measurements on recurring basis | Level III | CMBS | Minimum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale debt securities, term	0 years	0 years
Fair value measurements on recurring basis | Level III | CMBS | Minimum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0	0
Fair value measurements on recurring basis | Level III | CMBS | Maximum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale debt securities, term	7 years 7 months 6 days	7 years 7 months 6 days
Fair value measurements on recurring basis | Level III | CMBS | Maximum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	298.5	536.6
Fair value measurements on recurring basis | Level III | CMBS | Weighted-average
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale debt securities, term	6 years	5 years 3 months 18 days
Fair value measurements on recurring basis | Level III | CMBS | Weighted-average | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	5.9	7.1
Fair value measurements on recurring basis | Level III | Domestic Servicing Rights
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Domestic servicing rights	$ 12,406,000	$ 13,202,000
Servicing Asset, Valuation Technique [Extensible List]	us-gaap:ValuationTechniqueDiscountedCashFlowMember	us-gaap:ValuationTechniqueDiscountedCashFlowMember
Fair value measurements on recurring basis | Level III | Domestic Servicing Rights | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Servicing asset, measurement input	7.25
Fair value measurements on recurring basis | Level III | Domestic Servicing Rights | Discount rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Servicing asset, measurement input	15	15
Fair value measurements on recurring basis | Level III | Domestic Servicing Rights | Weighted-average | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Servicing asset, measurement input	7.25	7.50
Fair value measurements on recurring basis | Level III | Domestic Servicing Rights | Weighted-average | Discount rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Servicing asset, measurement input	15	15
Fair value measurements on recurring basis | Level III | VIE Assets | Minimum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE duration (in years)	0 years	0 years
Fair value measurements on recurring basis | Level III | VIE Assets | Minimum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE assets, measurement input	0	0
Fair value measurements on recurring basis | Level III | VIE Assets | Maximum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE duration (in years)	20 years 7 months 6 days	16 years 3 months 18 days
Fair value measurements on recurring basis | Level III | VIE Assets | Maximum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE assets, measurement input	752.4	312.2
Fair value measurements on recurring basis | Level III | VIE Assets | Weighted-average
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE duration (in years)	3 years 9 months 18 days	3 years 9 months 18 days
Fair value measurements on recurring basis | Level III | VIE Assets | Weighted-average | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE assets, measurement input	16.9	14.3
Fair value measurements on recurring basis | Level III | VIE liabilities | Minimum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE duration (in years)	0 years	0 years
Fair value measurements on recurring basis | Level III | VIE liabilities | Minimum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE liabilities, measurement input	0	0
Fair value measurements on recurring basis | Level III | VIE liabilities | Maximum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE duration (in years)	11 years	10 years 9 months 18 days
Fair value measurements on recurring basis | Level III | VIE liabilities | Maximum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE liabilities, measurement input	752.4	312.2
Fair value measurements on recurring basis | Level III | VIE liabilities | Weighted-average
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE duration (in years)		3 years 9 months 18 days
Fair value measurements on recurring basis | Level III | VIE liabilities | Weighted-average | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE liabilities, measurement input	17.3	14.4
Fair value measurements on recurring basis | Level III | VIE liabilities | Discounted cash flow | Weighted-average
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE duration (in years)	3 years 8 months 12 days
Primary beneficiary
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Total Assets	$ 62,367,110,000	$ 64,238,328,000
Total Liabilities	60,896,709,000	62,776,371,000
Primary beneficiary | Fair value measurements on recurring basis | VIE Assets
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Total Assets	62,367,110,000	64,238,328,000
Primary beneficiary | Fair value measurements on recurring basis | VIE liabilities
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Total Liabilities	60,896,709,000	62,776,371,000
Primary beneficiary | Fair value measurements on recurring basis | Level III | VIE Assets
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Total Assets	$ 62,367,110,000	$ 64,238,328,000
Derivative Asset, Valuation Technique [Extensible List]	us-gaap:ValuationTechniqueDiscountedCashFlowMember	us-gaap:ValuationTechniqueDiscountedCashFlowMember
Primary beneficiary | Fair value measurements on recurring basis | Level III | VIE liabilities
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Total Liabilities	$ 2,227,428,000	$ 2,019,876,000
Derivative Liability, Valuation Technique [Extensible List]	us-gaap:ValuationTechniqueDiscountedCashFlowMember	us-gaap:ValuationTechniqueDiscountedCashFlowMember