Fair Value of Financial Instruments - Level 3 Assets (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2016	Sep. 30, 2015	Sep. 30, 2016	Sep. 30, 2015
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Residential whole loans held at fair value excluded, purchase transaction not occurred	$ 92,800		$ 92,800
Recurring basis | Level 3 | Residential Whole Loans
Fair Value, Assets Measured on Recurring Basis, Unobservable Input Reconciliation, Calculation [Roll Forward]
Balance at beginning of period	684,582	$ 183,861	623,276	$ 143,472
Purchases and capitalized advances	50,071	357,258	169,830	410,981
Changes in fair value recorded in Net gain on residential whole loans held at fair value	10,913	2,362	25,529	3,872
Collection of principal, net of liquidation gains/losses	(18,119)	(7,152)	(48,909)	(15,716)
Transfer to REO	(22,985)	(4,792)	(65,264)	(11,072)
Balance at end of period	$ 704,462	$ 531,537	$ 704,462	$ 531,537