Residential Whole Loans - Narrative (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2016	Sep. 30, 2015	Sep. 30, 2016	Sep. 30, 2015	Jun. 30, 2016	Dec. 31, 2015	Jun. 30, 2015	Dec. 31, 2014
Investments, Debt and Equity Securities [Abstract]
Residential whole loans	$ 1,300,000		$ 1,300,000			$ 895,100
Equity interest in certificates (percent)	100.00%		100.00%
Residential whole loans at carrying value	$ 550,623		$ 550,623			271,845
Allowance for loan losses, residential whole loan at carrying value	1,023	$ 922	1,023	$ 922	$ 1,023	$ 1,165	$ 587	$ 137
(Reversal of)/Provision for loan losses, residential whole loan at carrying value	$ 0	$ 335	$ (142)	$ 785