Fair Value of Financial Instruments - Summary of quantitative information (Details) $ in Thousands	Dec. 31, 2021 USD ($)	Dec. 31, 2020 USD ($)
Residential Whole Loans | Level III
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Fair Value	$ 1,023,502
Residential Whole Loans | Level III | Discounted Cash Flow | Market Discount Rate | Minimum
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, measurement input	0.026	0.021
Residential Whole Loans | Level III | Discounted Cash Flow | Market Discount Rate | Maximum
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, measurement input	0.075	0.075
Residential Whole Loans | Level III | Discounted Cash Flow | Market Discount Rate | Weighted Average
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, measurement input	0.035	0.041
Residential Whole Loans | Level III | Discounted Cash Flow | Weighted Average Life | Minimum
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, term	1 year 4 months 24 days	1 year 6 months
Residential Whole Loans | Level III | Discounted Cash Flow | Weighted Average Life | Maximum
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, term	8 years 10 months 24 days	8 years 4 months 24 days
Residential Whole Loans | Level III | Discounted Cash Flow | Weighted Average Life | Weighted Average
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, term	3 years 1 month 6 days	2 years 10 months 24 days
Residential Bridge Loans | Level III
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Fair Value	$ 5,428
Residential Bridge Loans | Level III | Discounted Cash Flow | Market Discount Rate | Minimum
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, measurement input	0.098	0.080
Residential Bridge Loans | Level III | Discounted Cash Flow | Market Discount Rate | Maximum
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, measurement input	0.231	0.352
Residential Bridge Loans | Level III | Discounted Cash Flow | Market Discount Rate | Weighted Average
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, measurement input	0.172	0.180
Residential Bridge Loans | Level III | Discounted Cash Flow | Weighted Average Life | Minimum
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, term	3 months 18 days	3 months 18 days
Residential Bridge Loans | Level III | Discounted Cash Flow | Weighted Average Life | Maximum
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, term	3 years 7 months 6 days	2 years 7 months 6 days
Residential Bridge Loans | Level III | Discounted Cash Flow | Weighted Average Life | Weighted Average
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, term	2 years 4 months 24 days	1 year 3 months 18 days
Commercial Loans
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Fair Value	$ 130,572
Commercial Loans | Level III
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Fair Value	$ 130,572	$ 310,523
Commercial Loans | Level III | Discounted Cash Flow | Market Discount Rate | Minimum
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, measurement input	0.045	0.063
Commercial Loans | Level III | Discounted Cash Flow | Market Discount Rate | Maximum
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, measurement input	0.217	0.184
Commercial Loans | Level III | Discounted Cash Flow | Market Discount Rate | Weighted Average
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, measurement input	0.093	0.105
Commercial Loans | Level III | Discounted Cash Flow | Weighted Average Life | Minimum
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, term	6 months	6 months
Commercial Loans | Level III | Discounted Cash Flow | Weighted Average Life | Maximum
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, term	2 years 9 months 18 days	1 year 10 months 24 days
Commercial Loans | Level III | Discounted Cash Flow | Weighted Average Life | Weighted Average
Fair Value Measurement Inputs and Valuation Techniques [Line Items]
Loans held-for-sale, term	1 year 2 months 12 days	8 months 12 days