Residential Whole Loans and Bridge Loans Carrying Value Components (Details) - USD ($) $ in Thousands	Dec. 31, 2019	Dec. 31, 2018
Variable Interest Entity [Line Items]
Principal balance	$ 2,676,231
Residential Whole Loans, at Fair Value
Variable Interest Entity [Line Items]
Principal balance	1,325,443	$ 1,023,524
Unamortized premium	28,588	17,629
Unamortized discount	(2,839)	(3,145)
Amortized cost	1,351,192	1,038,008
Gross unrealized gains	26,363	7,573
Gross unrealized losses	(1,695)	(3,696)
Fair value	1,375,860	1,041,885
Residential Bridge Loans, at Fair Value
Variable Interest Entity [Line Items]
Principal balance	34,041	212,491
Unamortized premium	79	1,164
Unamortized discount	(13)	(316)
Amortized cost	34,107	213,339
Gross unrealized gains	10	212
Gross unrealized losses	(848)	(1,552)
Fair value	33,269	211,999
Residential Bridge Loans, at Amortized Cost
Variable Interest Entity [Line Items]
Principal balance	3,155	9,766
Unamortized premium	6	16
Unamortized discount	(11)	(62)
Amortized cost	$ 3,150	$ 9,720