Real Estate Securities - Components of Carrying Value (Which Equals Fair Value) of Residential Available for Sale Securities (Details) - USD ($) $ in Thousands	Dec. 31, 2017	Dec. 31, 2016
Schedule of Available-for-sale Securities [Line Items]
Carrying Value	$ 507,666	$ 572,752
Senior IO Securities
Schedule of Available-for-sale Securities [Line Items]
Carrying Value	140,989	136,546
Re-REMIC
Schedule of Available-for-sale Securities [Line Items]
Carrying Value	38,875	85,479
Subordinate Securities
Schedule of Available-for-sale Securities [Line Items]
Carrying Value	327,802	350,727
Residential
Schedule of Available-for-sale Securities [Line Items]
Principal balance	608,145	700,829
Credit reserve	(46,549)	(47,473)
Unamortized discount, net	(183,753)	(198,112)
Amortized cost	377,843	455,244
Gross unrealized gains	131,190	120,677
Gross unrealized losses	(1,367)	(3,169)
Carrying Value	507,666	572,752
Residential | Senior IO Securities
Schedule of Available-for-sale Securities [Line Items]
Principal balance	144,512	148,862
Credit reserve	(2,936)	(4,814)
Unamortized discount, net	(34,379)	(41,877)
Amortized cost	107,197	102,171
Gross unrealized gains	35,027	36,304
Gross unrealized losses	(1,235)	(1,929)
Carrying Value	140,989	136,546
Residential | Re-REMIC
Schedule of Available-for-sale Securities [Line Items]
Principal balance	44,613	95,608
Credit reserve	(5,820)	(6,857)
Unamortized discount, net	(9,662)	(19,613)
Amortized cost	29,131	69,138
Gross unrealized gains	9,744	16,341
Gross unrealized losses	0	0
Carrying Value	38,875	85,479
Residential | Subordinate Securities
Schedule of Available-for-sale Securities [Line Items]
Principal balance	419,020	456,359
Credit reserve	(37,793)	(35,802)
Unamortized discount, net	(139,712)	(136,622)
Amortized cost	241,515	283,935
Gross unrealized gains	86,419	68,032
Gross unrealized losses	(132)	(1,240)
Carrying Value	$ 327,802	$ 350,727