Residential Whole Loans-Fair Value Components of Net Income (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2015	Sep. 30, 2015	Jun. 30, 2015	Mar. 31, 2015	Dec. 31, 2014	Sep. 30, 2014	Jun. 30, 2014	Mar. 31, 2014	Dec. 31, 2015	Dec. 31, 2014	Dec. 31, 2013
Residential Whole Loans [Abstract]
Coupon payments and other income received									$ 9,303	$ 504
Net unrealized gains									6,540	(427)
Net gain on payoff/liquidation of loans									1,879	39
Total	$ 6,899	$ 5,565	$ 3,224	$ 2,034	$ 116	$ 0	$ 0	$ 0	$ 17,722	$ 116	$ 0