Residential Whole Loans and Bridge Loans - Summary of the Assets and Liabilities (Details) - USD ($) $ in Thousands		Dec. 31, 2021	Dec. 31, 2020	Dec. 31, 2019
Variable Interest Entity [Line Items]
Cash and cash equivalents		$ 40,193	$ 31,613	$ 31,331
Residential Whole Loans, at fair value ($1,023,502 and $1,008,782 pledged as collateral, at fair value, respectively)		1,023,502	1,008,782
Residential Bridge Loans ($5,428 and $12,813 at fair value and $5,207 and $12,960 pledged as collateral, respectively)		5,207	12,960
Investment related receivable		22,133	30,576
Interest receivable		11,823	13,568
Other assets		45,364	3,152
Total assets	[1]	2,825,700	3,336,009
Securitized debt, at fair value		1,863,488	2,446,012
Interest payable		10,272	12,006
Accounts payable and accrued expenses		4,842	2,686
Total liabilities	[2]	2,621,371	3,080,895
VIE
Variable Interest Entity [Line Items]
Cash and cash equivalents		266	0
Residential Whole Loans, at fair value ($1,023,502 and $1,008,782 pledged as collateral, at fair value, respectively)		1,023,502	1,008,782
Residential Bridge Loans ($5,428 and $12,813 at fair value and $5,207 and $12,960 pledged as collateral, respectively)		5,207	12,960
Investment related receivable		22,087	27,987
Interest receivable		10,572	10,936
Other assets		0	80
Total assets		2,432,064	2,810,678
Securitized debt, at fair value		1,863,488	2,446,012
Interest payable		6,480	7,882
Accounts payable and accrued expenses		78	89
Total liabilities		1,870,306	2,530,115
Residential Whole-Loan And Residential Bridge Loan | VIE
Variable Interest Entity [Line Items]
Cash and cash equivalents		266	0
Residential Whole Loans, at fair value ($1,023,502 and $1,008,782 pledged as collateral, at fair value, respectively)		1,023,502	1,008,782
Residential Bridge Loans ($5,428 and $12,813 at fair value and $5,207 and $12,960 pledged as collateral, respectively)		5,207	12,960
Investment related receivable		22,087	27,987
Interest receivable		5,282	4,688
Other assets		0	80
Total assets		1,056,344	1,054,497
Securitized debt, at fair value		519,118	892,290
Interest payable		1,316	2,222
Accounts payable and accrued expenses		69	77
Total liabilities		$ 520,503	$ 894,589

[1]  December 31, 2021December 31, 2020 (1) Assets of consolidated VIEs included in the total assets above: Cash and cash equivalents$266 $— Restricted cash260 76,132  Residential Whole Loans, at fair value ($1,023,502 and $1,008,782 pledged as collateral, at fair value, respectively) 1,023,502 1,008,782  Residential Bridge Loans ($5,207 and $11,858 at fair value and $5,207 and $12,960 pledged as collateral, respectively) 5,207 12,960 Securitized commercial loans, at fair value1,355,808 1,605,335  Commercial Loans, at fair value ($14,362 and $68,466 pledged as collateral, at fair value, respectively) 14,362 68,466 Investment related receivable22,087 27,987 Interest receivable10,572 10,936 Other assets— 80 Total assets of consolidated VIEs$2,432,064 $2,810,678 

[2] Liabilities of consolidated VIEs included in the total liabilities above: Securitized debt, net ($1,344,370 and $1,553,722 at fair value and $180,116 and $215,753 held by affiliates, respectively) $1,863,488 $2,446,012  Interest payable (includes $699 and $784 on securitized debt held by affiliates, respectively) 6,480 7,882 Accounts payable and accrued expenses78 89 Other liabilities260 76,132 Total liabilities of consolidated VIEs$1,870,306 $2,530,115