Fair Value of Financial Instruments - Summary of quantitative information (Details)	Mar. 31, 2020 USD ($)	Dec. 31, 2019 USD ($)
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Delinquent bridge loans, yield to maturity	100.00%
Residential  Whole Loans
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Fair value	$ 1,309,795,000	$ 1,375,860,000
Residential  Whole Loans | Discounted Cash Flow | Level III
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Fair value	1,161,199,000	1,200,566,000
Residential  Whole Loans | To-Be-Announced Pricing Model (TBA)
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Fair value	$ 148,600,000	$ 175.3
Residential  Whole Loans | Minimum | Discounted Cash Flow | Level III | Measurement Input, Discount Rate
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, measurement input	0.047	0.034
Residential  Whole Loans | Minimum | Discounted Cash Flow | Level III | Measurement Input, Expected Term
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, term	1 year 2 months 12 days	1 year 4 months 24 days
Residential  Whole Loans | Minimum | To-Be-Announced Pricing Model (TBA)
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Fair value		$ 101.39
Residential  Whole Loans | Maximum | Discounted Cash Flow | Level III | Measurement Input, Discount Rate
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, measurement input	0.089	0.070
Residential  Whole Loans | Maximum | Discounted Cash Flow | Level III | Measurement Input, Expected Term
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, term	6 years 7 months 6 days	7 years 9 months 18 days
Residential  Whole Loans | Maximum | To-Be-Announced Pricing Model (TBA)
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Fair value		$ 107.63
Residential  Whole Loans | Weighted Average | Discounted Cash Flow | Level III | Measurement Input, Discount Rate
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, measurement input	0.066	0.037
Residential  Whole Loans | Weighted Average | Discounted Cash Flow | Level III | Measurement Input, Expected Term
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, term	2 years 6 months	3 years
Residential Bridge Loans | Discounted Cash Flow | Level III
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Fair value	$ 26,050,000	$ 33,269,000
Residential Bridge Loans | Minimum | Discounted Cash Flow | Level III | Measurement Input, Discount Rate
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, measurement input	0.085	0.075
Residential Bridge Loans | Minimum | Discounted Cash Flow | Level III | Measurement Input, Expected Term
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, term	2 months 12 days	3 months 18 days
Residential Bridge Loans | Maximum | Discounted Cash Flow | Level III | Measurement Input, Discount Rate
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, measurement input	0.263	0.270
Residential Bridge Loans | Maximum | Discounted Cash Flow | Level III | Measurement Input, Expected Term
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, term	1 year 9 months 18 days	1 year 9 months 18 days
Residential Bridge Loans | Weighted Average | Discounted Cash Flow | Level III | Measurement Input, Discount Rate
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, measurement input	0.116	0.098
Residential Bridge Loans | Weighted Average | Discounted Cash Flow | Level III | Measurement Input, Expected Term
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, term	9 months 18 days	9 months 18 days
Commercial Loans
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Fair value	$ 320,308,000
Commercial Loans | Discounted Cash Flow | Level III
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Fair value	$ 320,308,000	$ 370,213,000
Commercial Loans | Minimum | Discounted Cash Flow | Level III | Measurement Input, Discount Rate
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, measurement input	0.065	0.047
Commercial Loans | Minimum | Discounted Cash Flow | Level III | Measurement Input, Expected Term
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, term	1 year 2 months 12 days	4 months 24 days
Commercial Loans | Maximum | Discounted Cash Flow | Level III | Measurement Input, Discount Rate
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, measurement input	0.149	0.109
Commercial Loans | Maximum | Discounted Cash Flow | Level III | Measurement Input, Expected Term
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, term	2 years 8 months 12 days	2 years 10 months 24 days
Commercial Loans | Weighted Average | Discounted Cash Flow | Level III | Measurement Input, Discount Rate
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, measurement input	0.094	0.075
Commercial Loans | Weighted Average | Discounted Cash Flow | Level III | Measurement Input, Expected Term
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, term	1 year 4 months 24 days	1 year 7 months 6 days