Real Estate Securities - Changes of Unamortized Discount and Designated Credit Reserves on Residential Available for Sale Securities (Details) - USD ($) $ in Thousands	3 Months Ended	9 Months Ended
	Sep. 30, 2020	Sep. 30, 2020
Credit Reserve
Beginning balance	$ 1,471	$ 0
Sales, calls, other	0	0
Ending Balance	1,027	1,027
Unamortized Discount, Net
Beginning balance		124,255
Ending Balance	99,221	99,221
Residential
Credit Reserve
Beginning balance	37,785	32,940
Amortization of net discount	0	0
Realized credit losses	(194)	(897)
Acquisitions	1,303	6,487
Sales, calls, other	0	(726)
(Release of) transfers to credit reserves, net	4,292	5,382
Ending Balance	43,186	43,186
Unamortized Discount, Net
Beginning balance	104,260	124,255
Amortization of net discount	(1,766)	(4,607)
Realized credit losses	0	0
Acquisitions	1,019	1,796
Sales, calls, other	0	(16,841)
(Release of) transfers to credit reserves, net	(4,292)	(5,382)
Ending Balance	$ 99,221	$ 99,221