Real Estate Securities - Changes of Unamortized Discount and Designated Credit Reserves on Residential Available for Sale Securities (Details) - Residential - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2017	Dec. 31, 2016
Credit Reserve
Beginning balance	$ 47,473	$ 48,869
Amortization of net discount	0	0
Realized credit losses	(4,187)	(5,830)
Acquisitions	9,118	9,311
Sales, calls, other	(3,404)	(4,968)
Impairments	1,011	368
Transfers to (release of) credit reserves, net	(3,462)	(277)
Ending Balance	46,549	47,473
Unamortized Discount, Net
Beginning balance	198,112	237,107
Amortization of net discount	(18,795)	(26,253)
Realized credit losses	0	0
Acquisitions	13,080	11,461
Sales, calls, other	(12,106)	(24,480)
Impairments	0	0
Transfers to (release of) credit reserves, net	3,462	277
Ending Balance	$ 183,753	$ 198,112