Residential Whole-Loans and Bridge Loans - Summary of the assets and liabilities of the residential and commercial loan trusts (Details) - USD ($) $ in Thousands	Mar. 31, 2020	Dec. 31, 2019	Mar. 31, 2019
Variable Interest Entity [Line Items]
Cash and cash equivalents	$ 10,342	$ 31,331	$ 19,558
Residential Whole Loans, at fair value ($1,309,795 and $1,375,860 pledged as collateral, at fair value, respectively)	1,309,795	1,375,860
Residential Bridge Loans ($24,987 and $31,748 at fair value and $27,571 and $34,897 pledged as collateral, respectively)	27,571	34,897
Investment related receivable	72,826	19,931
Interest receivable	14,805	19,413
Other assets	5,697	4,509
Total assets	3,203,583	5,160,971
Securitized debt, net ($396,824 and $681,643 at fair value and $53,527 and $142,905 held by affiliates, respectively)	1,139,121	1,477,454
Interest payable (includes $536 and $647 on securitized debt held by affiliates, respectively)	6,429	15,001
Accounts payable and accrued expenses	6,307	3,188
Other liabilities	45,779	52,948
Total liabilities	3,021,350	4,596,510
Fair value of residential whole-loans pledged as collateral	1,309,795	1,375,860
Residential bridge loan, at fair value	26,050	33,269
VIE
Variable Interest Entity [Line Items]
Cash and cash equivalents	4,542	7,589
Residential Whole Loans, at fair value ($1,309,795 and $1,375,860 pledged as collateral, at fair value, respectively)	1,309,795	1,375,860
Residential Bridge Loans ($24,987 and $31,748 at fair value and $27,571 and $34,897 pledged as collateral, respectively)	27,571	34,897
Investment related receivable	24,738	19,138
Interest receivable	10,226	10,829
Other assets	101	90
Total assets	1,959,017	2,501,179
Securitized debt, net ($396,824 and $681,643 at fair value and $53,527 and $142,905 held by affiliates, respectively)	1,139,121	1,477,454
Interest payable (includes $536 and $647 on securitized debt held by affiliates, respectively)	3,215	3,886
Accounts payable and accrued expenses	128	185
Other liabilities	33,229	52,948
Total liabilities	1,175,693	1,534,473
Fair value of residential whole-loans pledged as collateral	1,309,795	1,375,860
Residential bridge loan, at fair value	24,987	31,748
Residential Whole-Loan And Residential Bridge Loan | VIE
Variable Interest Entity [Line Items]
Cash and cash equivalents	4,542	1,811
Residential Whole Loans, at fair value ($1,309,795 and $1,375,860 pledged as collateral, at fair value, respectively)	1,309,795	1,375,860
Residential Bridge Loans ($24,987 and $31,748 at fair value and $27,571 and $34,897 pledged as collateral, respectively)	27,571	34,897
Investment related receivable	24,738	19,138
Interest receivable	8,517	7,840
Other assets	101	90
Total assets	1,375,264	1,439,636
Securitized debt, net ($396,824 and $681,643 at fair value and $53,527 and $142,905 held by affiliates, respectively)	742,297	795,811
Interest payable (includes $536 and $647 on securitized debt held by affiliates, respectively)	2,210	2,367
Accounts payable and accrued expenses	122	173
Total liabilities	744,629	798,351
Fair value of residential whole-loans pledged as collateral	1,309,795	1,375,860
Residential bridge loan, at fair value	$ 24,987	$ 31,748