Fair Value of Financial Instruments - Fair Value vs Unpaid Principal Balance (Details) - USD ($) $ in Thousands	Sep. 30, 2016	Dec. 31, 2015
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total Residential Whole Loans, Fair Value	$ 797,191	$ 623,276
Total Residential Whole Loans, Unpaid Principal Balance	840,964	786,330
Residential whole loans held at fair value excluded, purchase transaction not occurred	92,800
Residential Whole Loans
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total Residential Whole Loans, Fair Value	704,462	623,276
Total Residential Whole Loans, Unpaid Principal Balance	840,964	786,330
Total Residential Whole Loans, Difference	(136,502)	(163,054)
Residential Whole Loans 90 days or more past due, Fair Value	504,337	493,640
Residential Whole Loans 90 days or more past due, Unpaid Principal Balance	613,824	637,459
Residential Whole Loans 90 days or more past due, Difference	$ (109,487)	$ (143,819)