Residential Whole Loans and Bridge Loans Summary of the Assets and Liabilities (Details) - USD ($) $ in Thousands	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Variable Interest Entity [Line Items]
Cash and cash equivalents	$ 31,331	$ 21,987	$ 48,024
Residential Whole-Loans, at fair value ($1,375,860 and $1,041,885 pledged as collateral, at fair value, respectively)	1,375,860	1,041,885
Commercial Loans, at fair value	350,213	196,123
Investment related receivable	19,931	42,945
Interest receivable	19,413	21,959
Other assets	4,509	2,488
Total assets	5,160,971	4,497,395
Securitized debt, net ($681,643 and $949,626 at fair value and $142,905 and $246,802 held by affiliates, respectively)	1,477,454	949,626
Interest payable	15,001	8,532
Accounts payable and accrued expenses	3,188	3,858
Other liabilities	52,948	56,031
Total liabilities	4,596,510	3,994,386
VIE
Variable Interest Entity [Line Items]
Cash and cash equivalents	7,589	674
Residential Whole-Loans, at fair value ($1,375,860 and $1,041,885 pledged as collateral, at fair value, respectively)		1,041,885
Residential Bridge Loans ($31,748 and $211,766 at fair value and $34,897 and $221,486 pledged as collateral, respectively)	34,897	221,486
Commercial Loans, at fair value	90,788	196,123
Investment related receivable		42,945
Interest receivable	10,829	15,540
Other assets	90	178
Total assets	2,501,179	2,588,150
Securitized debt, net ($681,643 and $949,626 at fair value and $142,905 and $246,802 held by affiliates, respectively)	1,477,454	949,626
Interest payable	3,886	2,419
Accounts payable and accrued expenses	185	708
Other liabilities	52,948	56,033
Total liabilities	1,534,473	1,008,786
Residential Whole-Loan And Residential Bridge Loan | VIE
Variable Interest Entity [Line Items]
Cash and cash equivalents	1,811	674
Residential Whole-Loans, at fair value ($1,375,860 and $1,041,885 pledged as collateral, at fair value, respectively)	1,375,860	1,041,885
Residential Bridge Loans ($31,748 and $211,766 at fair value and $34,897 and $221,486 pledged as collateral, respectively)	34,897	221,486
Commercial Loans, at fair value	0	30,000
Investment related receivable	19,138	42,945
Interest receivable	7,840	11,807
Other assets	90	178
Total assets	1,439,636	1,348,975
Securitized debt, net ($681,643 and $949,626 at fair value and $142,905 and $246,802 held by affiliates, respectively)	795,811	0
Interest payable	2,367	0
Accounts payable and accrued expenses	173	677
Other liabilities	0	225
Total liabilities	$ 798,351	$ 902