Residential Whole Loans (Carrying Value Accretable Yield Rollforward) (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2019	Jun. 30, 2018	Jun. 30, 2019	Jun. 30, 2018
Residential Whole Loans Accretable Yield [Roll Forward]
Balance at beginning of period	$ 398,958	$ 413,404	$ 415,330	$ 421,872
Accretion	(10,997)	(10,910)	(22,396)	(21,941)
Liquidations and other	(11,808)	(12,840)	(22,296)	(15,010)
Reclassifications from non-accretable difference, net	111	11,421	5,626	16,154
Balance at end of period	$ 376,264	401,075	$ 376,264	401,075
Purchase credit impaired loans transaction closed in current period which were purchased in prior periods		$ 57,600		$ 57,600