Fair Value of Financial Instruments - Summary of quantitative information (Details)	Dec. 31, 2019 USD ($)	Dec. 31, 2018 USD ($)
Residential Whole Loans
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Fair value	$ 1,200,566,000	$ 1,041,885,000
Residential Whole Loans | To-Be-Announced Pricing Model (TBA)
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Fair value	175,294
Residential Whole Loans | To-Be-Announced Pricing Model (TBA) | Minimum
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans Held-for-sale, Measurement Input, Pricing	101.39
Residential Whole Loans | To-Be-Announced Pricing Model (TBA) | Maximum
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans Held-for-sale, Measurement Input, Pricing	107.63
Residential Whole Loans | Level III
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Fair value	$ 1,375,860,000
Residential Whole Loans | Level III | Discounted Cash Flow | Measurement Input, Discount Rate | Minimum
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, measurement input	0.034	0.035
Residential Whole Loans | Level III | Discounted Cash Flow | Measurement Input, Discount Rate | Maximum
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, measurement input	0.070	0.079
Residential Whole Loans | Level III | Discounted Cash Flow | Measurement Input, Discount Rate | Weighted Average
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, measurement input	0.037	0.055
Residential Whole Loans | Level III | Discounted Cash Flow | Measurement Input, Expected Term | Minimum
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, term	1 year 4 months 24 days	9 months 18 days
Residential Whole Loans | Level III | Discounted Cash Flow | Measurement Input, Expected Term | Maximum
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, term	7 years 9 months 18 days	10 years 3 months 18 days
Residential Whole Loans | Level III | Discounted Cash Flow | Measurement Input, Expected Term | Weighted Average
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, term	3 years	2 years 9 months 18 days
Residential Bridge Loans
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Fair value	$ 33,269,000	$ 211,999,000
Residential Bridge Loans | Level III | Discounted Cash Flow | Measurement Input, Discount Rate | Minimum
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, measurement input	0.075	0.056
Residential Bridge Loans | Level III | Discounted Cash Flow | Measurement Input, Discount Rate | Maximum
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, measurement input	0.270	1.453
Residential Bridge Loans | Level III | Discounted Cash Flow | Measurement Input, Discount Rate | Weighted Average
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, measurement input	0.098	0.113
Residential Bridge Loans | Level III | Discounted Cash Flow | Measurement Input, Expected Term | Minimum
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, term	3 months 18 days	1 month 6 days
Residential Bridge Loans | Level III | Discounted Cash Flow | Measurement Input, Expected Term | Maximum
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, term	1 year 9 months 18 days	1 year 7 months 6 days
Residential Bridge Loans | Level III | Discounted Cash Flow | Measurement Input, Expected Term | Weighted Average
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, term	9 months 18 days	6 months
Commercial Loans
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Fair value	$ 370,213,000	$ 216,123,000
Commercial Loans | Level III
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Fair value		$ 216,123,000
Commercial Loans | Level III | Discounted Cash Flow | Measurement Input, Discount Rate | Minimum
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, measurement input	0.047	0.067
Commercial Loans | Level III | Discounted Cash Flow | Measurement Input, Discount Rate | Maximum
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, measurement input	0.109	0.092
Commercial Loans | Level III | Discounted Cash Flow | Measurement Input, Discount Rate | Weighted Average
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, measurement input	0.075	0.076
Commercial Loans | Level III | Discounted Cash Flow | Measurement Input, Expected Term | Minimum
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, term	4 months 24 days	10 months 24 days
Commercial Loans | Level III | Discounted Cash Flow | Measurement Input, Expected Term | Maximum
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, term	2 years 10 months 24 days	2 years 8 months 12 days
Commercial Loans | Level III | Discounted Cash Flow | Measurement Input, Expected Term | Weighted Average
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, term	1 year 7 months 6 days	2 years 1 month 6 days