Fair Value of Financial Instruments - Level 3 (Details) - Residential whole loans - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2015	Dec. 31, 2014
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at beginning of period	$ 143,472	$ 0
Purchases and capitalized advances	534,574	147,471
Changes in fair value recorded in Net gain on residential whole loans held at fair value	6,539	(388)
Collection of principal, net of liquidation gains/losses	(34,767)	(1,038)
Transfer to REO	(26,542)	(2,573)
Balance at end of period	$ 623,276	$ 143,472