Real Estate Securities - Components of Carrying Value (Which Equals Fair Value) of Residential Available for Sale Securities (Details) - USD ($) $ in Thousands	Dec. 31, 2019	Dec. 31, 2018
Debt Securities, Available-for-sale [Line Items]
Principal balance	$ 304,077	$ 431,112
Credit reserve	(32,940)	(41,370)
Unamortized discount, net	(124,255)	(151,200)
Amortized cost	146,882	238,542
Gross unrealized gains	92,481	95,888
Gross unrealized losses	(29)	(548)
Carrying Value	239,334	333,882
Senior IO Securities
Debt Securities, Available-for-sale [Line Items]
Principal balance	26,331	91,736
Credit reserve	(533)	(7,790)
Unamortized discount, net	(10,427)	(18,460)
Amortized cost	15,371	65,486
Gross unrealized gains	10,450	22,178
Gross unrealized losses	(29)	(49)
Carrying Value	25,792	87,615
Mezzanine
Debt Securities, Available-for-sale [Line Items]
Principal balance	13,512	36,852
Credit reserve	0	0
Unamortized discount, net	(527)	(3,697)
Amortized cost	12,985	33,155
Gross unrealized gains	702	3,252
Gross unrealized losses	0	0
Carrying Value	13,687	36,407
Subordinate Securities
Debt Securities, Available-for-sale [Line Items]
Principal balance	264,234	302,524
Credit reserve	(32,407)	(33,580)
Unamortized discount, net	(113,301)	(129,043)
Amortized cost	118,526	139,901
Gross unrealized gains	81,329	70,458
Gross unrealized losses	0	(499)
Carrying Value	$ 199,855	$ 209,860