Residential Whole Loans - Narrative (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2015	Sep. 30, 2015	Jun. 30, 2015	Mar. 31, 2015	Dec. 31, 2014	Sep. 30, 2014	Jun. 30, 2014	Mar. 31, 2014	Dec. 31, 2015	Dec. 31, 2014	Dec. 31, 2013
Residential Whole Loans [Abstract]
Residential Whole Loans	$ 895,100				$ 351,400				$ 895,100	$ 351,400
Equity interest in certificates (percent)	100.00%								100.00%
Residential whole loans at carrying value	$ 271,845				207,923				$ 271,845	207,923
Provisions for loan losses									1,028	137
Net gain on residential whole loans held at fair value	$ 6,899	$ 5,565	$ 3,224	$ 2,034	$ 116	$ 0	$ 0	$ 0	$ 17,722	$ 116	$ 0