Real Estate Securities - Changes of Unamortized Discount and Designated Credit Reserves on Residential Available for Sale Securities (Details) - USD ($) $ in Thousands	3 Months Ended	6 Months Ended
	Jun. 30, 2020	Jun. 30, 2020
Credit Reserve
Beginning balance	$ 1,525	$ 0
Sales, calls, other	0	0
Ending Balance	1,471	1,471
Unamortized Discount, Net
Beginning balance		124,255
Ending Balance	104,260	104,260
Residential
Credit Reserve
Beginning balance	37,717	32,940
Amortization of net discount	0	0
Realized credit losses	(184)	(703)
Acquisitions	0	5,184
Sales, calls, other	(520)	(726)
(Release of) transfers to credit reserves, net	772	1,090
Ending Balance	37,785	37,785
Unamortized Discount, Net
Beginning balance	109,538	124,255
Amortization of net discount	(1,087)	(2,841)
Realized credit losses	0	0
Acquisitions	0	777
Sales, calls, other	(3,419)	(16,841)
(Release of) transfers to credit reserves, net	(772)	(1,090)
Ending Balance	$ 104,260	$ 104,260