Fair Value of Financial Instruments-Fair Value vs Unpaid Principal Balance (Details) - USD ($) $ in Thousands	Jun. 30, 2015	Dec. 31, 2014
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total Residential Whole Loans, Fair Value	$ 183,861	$ 143,472
Total Residential Whole Loans, Unpaid Principal Balance	247,310	182,613
Residential whole loans
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Total Residential Whole Loans, Fair Value	183,861	143,472
Total Residential Whole Loans, Unpaid Principal Balance	247,310	182,613
Total Residential Whole Loans, Difference	(63,449)	(39,141)
Residential Whole Loans 90 days or more past due, Fair Value	130,207	128,591
Residential Whole Loans 90 days or more past due, Unpaid Principal Balance	184,685	165,358
Residential Whole Loans 90 days or more past due, Difference	$ (54,478)	$ (36,767)