Fair Value of Financial Instruments - Fair Value vs Unpaid Principal Balance (Details) - USD ($) $ in Thousands	Jun. 30, 2016	Dec. 31, 2015
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total Residential Whole Loans, Fair Value	$ 684,582	$ 623,276
Total Residential Whole Loans, Unpaid Principal Balance	844,846	786,330
Residential Whole Loans
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total Residential Whole Loans, Fair Value	684,582	623,276
Total Residential Whole Loans, Unpaid Principal Balance	844,846	786,330
Total Residential Whole Loans, Difference	(160,264)	(163,054)
Residential Whole Loans 90 days or more past due, Fair Value	520,590	493,640
Residential Whole Loans 90 days or more past due, Unpaid Principal Balance	653,151	637,459
Residential Whole Loans 90 days or more past due, Difference	$ (132,561)	$ (143,819)