Real Estate Securities - Changes of Unamortized Discount and Designated Credit Reserves on Residential Available for Sale Securities (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2020	Dec. 31, 2019
Credit Reserve
Beginning balance	$ 32,940
Ending Balance	44,967	$ 32,940
Unamortized Discount, Net
Beginning balance	124,255
Ending Balance	95,718	124,255
Residential
Credit Reserve
Beginning balance	32,940	41,370
Amortization of net discount	0	0
Realized credit losses	(2,282)	(2,606)
Acquisitions	7,248	3,712
Sales, calls, other	(731)	(9,453)
Impairments	0	0
Transfers to (release of) credit reserves, net	7,792	(83)
Ending Balance	44,967	32,940
Unamortized Discount, Net
Beginning balance	124,255	151,200
Amortization of net discount	(6,538)	(7,921)
Realized credit losses	0	0
Acquisitions	2,634	1,910
Sales, calls, other	(16,841)	(21,017)
Impairments	0	0
Transfers to (release of) credit reserves, net	(7,792)	83
Ending Balance	$ 95,718	$ 124,255