Real Estate Securities - Changes of Unamortized Discount and Designated Credit Reserves on Residential Available for Sale Securities (Details) - Residential - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2016	Dec. 31, 2015
Credit Reserve
Beginning balance	$ 48,869	$ 70,067
Amortization of net discount	0	0
Realized credit losses	(5,830)	(8,535)
Acquisitions	9,311	2,557
Sales, calls, other	(4,968)	(7,296)
Impairments	368	0
Transfers to (release of) credit reserves, net	(277)	(7,924)
Ending Balance	47,473	48,869
Available For Sale Securities Credit Reserve [Roll Forward]
Beginning balance	237,107	296,342
Amortization of net discount	(26,253)	(36,850)
Realized credit losses	0	0
Acquisitions	11,461	15,791
Sales, calls, other	(24,480)	(46,346)
Impairments	0	246
Transfers to (release of) credit reserves, net	277	7,924
Ending Balance	$ 198,112	$ 237,107