Residential Whole Loans (Fair Value Components of Net Income) (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2019	Jun. 30, 2018	Jun. 30, 2019	Jun. 30, 2018
Receivables [Abstract]
Coupon payments and other income received	$ 21,411	$ 19,002	$ 40,884	$ 34,400
Net unrealized (losses)/gains	21,188	4,599	20,128	18,346
Net gain on payoff/liquidation of loans	2,596	4,044	4,879	6,952
Net gain on transfers to REO	6,278	4,798	10,849	11,243
Total	$ 51,473	$ 32,443	$ 76,740	$ 70,941