Fair Value - Significant unobservable inputs (Details) $ in Thousands	3 Months Ended	12 Months Ended
	Mar. 31, 2020 USD ($)	Dec. 31, 2019 USD ($)
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Domestic servicing rights	$ 16,524	$ 16,917
Total Assets	77,577,189	78,042,336
Total Liabilities	72,759,838	72,905,322
Fair value measurements on recurring basis | Loans held-for-sale
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, fair value option	1,347,797	1,436,194
Fair value measurements on recurring basis | RMBS
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities	170,640	189,576
Fair value measurements on recurring basis | CMBS
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities	32,855	37,360
Fair value measurements on recurring basis | Domestic Servicing Rights
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Domestic servicing rights	16,524	16,917
Fair value measurements on recurring basis | Level III | Loans held-for-sale
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, fair value option	$ 1,347,797	$ 1,436,194
Loans Held-for-sale, Valuation Technique [Extensible List]	us-gaap:ValuationTechniqueDiscountedCashFlowMember	us-gaap:ValuationTechniqueDiscountedCashFlowMember
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Minimum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, duration	1 year 4 months 24 days	1 year 3 months 18 days
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Minimum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, measurement input	3.2	3.4
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Maximum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, duration	11 years 3 months 18 days	11 years 3 months 18 days
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Maximum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, measurement input	8.9	5.9
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Weighted-average
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, duration	4 years 9 months 18 days
Fair value measurements on recurring basis | Level III | Loans held-for-sale | Weighted-average | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loans held-for-sale, measurement input	5.2
Fair value measurements on recurring basis | Level III | RMBS
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities	$ 170,640	$ 189,576
Debt Securities, Available-for-sale, Valuation Technique [Extensible List]	us-gaap:ValuationTechniqueDiscountedCashFlowMember	us-gaap:ValuationTechniqueDiscountedCashFlowMember
Loss severity for specified percentage of portfolio (as a percent)	28.00%	34.00%
Fair value measurements on recurring basis | Level III | RMBS | Minimum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loss severity for specified percentage of portfolio (as a percent)	45.00%	45.00%
Fair value measurements on recurring basis | Level III | RMBS | Minimum | Constant prepayment rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	3.3	3.1
Fair value measurements on recurring basis | Level III | RMBS | Minimum | Constant default rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.4	0.5
Fair value measurements on recurring basis | Level III | RMBS | Minimum | Loss severity
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0	0
Fair value measurements on recurring basis | Level III | RMBS | Minimum | Delinquency rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	4	5
Fair value measurements on recurring basis | Level III | RMBS | Minimum | Servicer advances
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	27	27
Fair value measurements on recurring basis | Level III | RMBS | Minimum | Annual coupon deterioration
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0	0
Fair value measurements on recurring basis | Level III | RMBS | Minimum | Putback amount per projected total collateral loss
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0	0
Fair value measurements on recurring basis | Level III | RMBS | Maximum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Loss severity for specified percentage of portfolio (as a percent)	80.00%	80.00%
Fair value measurements on recurring basis | Level III | RMBS | Maximum | Constant prepayment rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	17.8	24.9
Fair value measurements on recurring basis | Level III | RMBS | Maximum | Constant default rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	4.8	5.0
Fair value measurements on recurring basis | Level III | RMBS | Maximum | Loss severity
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	82	93
Fair value measurements on recurring basis | Level III | RMBS | Maximum | Delinquency rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	(29)	29
Fair value measurements on recurring basis | Level III | RMBS | Maximum | Servicer advances
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	(85)	85
Fair value measurements on recurring basis | Level III | RMBS | Maximum | Annual coupon deterioration
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.9	1.6
Fair value measurements on recurring basis | Level III | RMBS | Maximum | Putback amount per projected total collateral loss
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	25	28
Fair value measurements on recurring basis | Level III | RMBS | Weighted-average | Constant prepayment rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	7.7
Fair value measurements on recurring basis | Level III | RMBS | Weighted-average | Constant default rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	2.4
Fair value measurements on recurring basis | Level III | RMBS | Weighted-average | Loss severity
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	29
Fair value measurements on recurring basis | Level III | RMBS | Weighted-average | Delinquency rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	17
Fair value measurements on recurring basis | Level III | RMBS | Weighted-average | Servicer advances
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	61
Fair value measurements on recurring basis | Level III | RMBS | Weighted-average | Annual coupon deterioration
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0.1
Fair value measurements on recurring basis | Level III | RMBS | Weighted-average | Putback amount per projected total collateral loss
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	2.9
Fair value measurements on recurring basis | Level III | CMBS
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities	$ 22,435	$ 25,008
Debt Securities, Available-for-sale, Valuation Technique [Extensible List]	us-gaap:ValuationTechniqueDiscountedCashFlowMember	us-gaap:ValuationTechniqueDiscountedCashFlowMember
Fair value measurements on recurring basis | Level III | CMBS | Minimum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale debt securities, term	0 years	0 years
Fair value measurements on recurring basis | Level III | CMBS | Minimum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	0	0
Fair value measurements on recurring basis | Level III | CMBS | Maximum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale debt securities, term	9 years 8 months 12 days	9 years 8 months 12 days
Fair value measurements on recurring basis | Level III | CMBS | Maximum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	6.2	122.9
Fair value measurements on recurring basis | Level III | CMBS | Weighted-average
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale debt securities, term	4 years 9 months 18 days
Fair value measurements on recurring basis | Level III | CMBS | Weighted-average | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Available-for-sale securities, measurement input	3.9
Fair value measurements on recurring basis | Level III | Domestic Servicing Rights
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Domestic servicing rights	$ 16,524	$ 16,917
Servicing Asset, Valuation Technique [Extensible List]	us-gaap:ValuationTechniqueDiscountedCashFlowMember	us-gaap:ValuationTechniqueDiscountedCashFlowMember
Fair value measurements on recurring basis | Level III | Domestic Servicing Rights | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Servicing asset, measurement input	7.75	7.50
Fair value measurements on recurring basis | Level III | Domestic Servicing Rights | Discount rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Servicing asset, measurement input	15	15
Fair value measurements on recurring basis | Level III | Domestic Servicing Rights | Minimum | Control migration
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Servicing asset, measurement input	0	0
Fair value measurements on recurring basis | Level III | Domestic Servicing Rights | Maximum | Control migration
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Servicing asset, measurement input	80	80
Fair value measurements on recurring basis | Level III | Domestic Servicing Rights | Weighted-average
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Servicing asset, measurement input	80
Fair value measurements on recurring basis | Level III | Domestic Servicing Rights | Weighted-average | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Servicing asset, measurement input	7.75
Fair value measurements on recurring basis | Level III | Domestic Servicing Rights | Weighted-average | Discount rate
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Servicing asset, measurement input	15
Fair value measurements on recurring basis | Level III | VIE Assets | Minimum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE duration (in years)	0 years	0 years
Fair value measurements on recurring basis | Level III | VIE Assets | Minimum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE assets, measurement input	0	0
Fair value measurements on recurring basis | Level III | VIE Assets | Maximum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE duration (in years)	18 years 2 months 12 days	19 years 2 months 12 days
Fair value measurements on recurring basis | Level III | VIE Assets | Maximum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE assets, measurement input	862.5	690.7
Fair value measurements on recurring basis | Level III | VIE Assets | Weighted-average
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE duration (in years)	5 years 1 month 6 days
Fair value measurements on recurring basis | Level III | VIE Assets | Weighted-average | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE assets, measurement input	10.7
Fair value measurements on recurring basis | Level III | VIE liabilities | Minimum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE duration (in years)	0 years	0 years
Fair value measurements on recurring basis | Level III | VIE liabilities | Minimum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE liabilities, measurement input	0	0
Fair value measurements on recurring basis | Level III | VIE liabilities | Maximum
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE duration (in years)	11 years 4 months 24 days	12 years 8 months 12 days
Fair value measurements on recurring basis | Level III | VIE liabilities | Maximum | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE liabilities, measurement input	862.5	690.7
Fair value measurements on recurring basis | Level III | VIE liabilities | Weighted-average
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE duration (in years)	5 years 2 months 12 days
Fair value measurements on recurring basis | Level III | VIE liabilities | Weighted-average | Debt yield
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
VIE liabilities, measurement input	10.6
Primary beneficiary
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Total Assets	$ 61,157,805	$ 62,187,175
Total Liabilities	59,807,306	60,743,494
Primary beneficiary | Fair value measurements on recurring basis | VIE Assets
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Total Assets	61,157,805	62,187,175
Primary beneficiary | Fair value measurements on recurring basis | VIE liabilities
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Total Liabilities	59,807,306	60,743,494
Primary beneficiary | Fair value measurements on recurring basis | Level III | VIE Assets
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Total Assets	$ 61,157,805	$ 62,187,175
Derivative Asset, Valuation Technique [Extensible List]	us-gaap:ValuationTechniqueDiscountedCashFlowMember	us-gaap:ValuationTechniqueDiscountedCashFlowMember
Primary beneficiary | Fair value measurements on recurring basis | Level III | VIE liabilities
Quantitative information for Level 3 Fair Value Measurements for assets and liabilities measured at fair value on recurring basis
Total Liabilities	$ 1,506,437	$ 2,537,392
Derivative Liability, Valuation Technique [Extensible List]	us-gaap:ValuationTechniqueDiscountedCashFlowMember	us-gaap:ValuationTechniqueDiscountedCashFlowMember