Residential Whole Loans - Fair Value Components of Net Income (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2016	Sep. 30, 2015	Sep. 30, 2016	Sep. 30, 2015
Investments, Debt and Equity Securities [Abstract]
Coupon payments and other income received	$ 6,253	$ 2,772	$ 15,987	$ 5,665
Net unrealized gains	10,913	2,362	25,529	3,872
Net gain on payoff/liquidation of loans	1,535	431	3,536	1,286
Total	$ 18,701	$ 5,565	$ 45,052	$ 10,823