Residential Whole-Loans and Bridge Loans - Components of the fair value of Residential Whole-Loans (Details) - USD ($) $ in Thousands	Mar. 31, 2020	Dec. 31, 2019
Residential  Whole Loans
Variable Interest Entity [Line Items]
Principal  Balance	$ 1,352,778	$ 1,325,443
Unamortized premium	30,983	28,588
Unamortized discount	(2,662)	(2,839)
Amortized cost	1,381,099	1,351,192
Gross unrealized gains	227	26,363
Gross unrealized losses	(71,531)	(1,695)
Fair value	1,309,795	1,375,860
Residential Bridge Loans, At Fair Value
Variable Interest Entity [Line Items]
Principal  Balance	27,059	34,041
Unamortized premium	52	79
Unamortized discount	(7)	(13)
Amortized cost	27,104	34,107
Gross unrealized gains	3	10
Gross unrealized losses	(1,057)	(848)
Fair value	26,050	33,269
Residential Bridge Loans, At Amortized Cost
Variable Interest Entity [Line Items]
Principal  Balance	2,591	3,155
Unamortized premium	3	6
Unamortized discount	(10)	(11)
Amortized cost	$ 2,584	$ 3,150