Residential Whole Loans-Fair Value Components of Net Income (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2016	Sep. 30, 2016	Jun. 30, 2016	Mar. 31, 2016	Dec. 31, 2015	Sep. 30, 2015	Jun. 30, 2015	Mar. 31, 2015	Dec. 31, 2016	Dec. 31, 2015	Dec. 31, 2014
Residential Whole Loans [Abstract]
Coupon payments and other income received									$ 23,017	$ 9,304	$ 504
Net unrealized gains/(losses)									31,254	6,539	(427)
Net gain on payoff/liquidation of loans									5,413	1,879	39
Total	$ 14,632	$ 18,701	$ 14,470	$ 11,881	$ 6,899	$ 5,565	$ 3,224	$ 2,034	$ 59,684	$ 17,722	$ 116