Residential Whole Loans - Fair Value (Details) $ in Thousands	9 Months Ended	12 Months Ended
	Sep. 30, 2016 USD ($) loan	Dec. 31, 2015 USD ($) loan
Investments, Debt and Equity Securities [Abstract]
Outstanding principal balance	$ 840,964	$ 786,330
Aggregate fair value	$ 704,462	$ 623,276
Number of loans | loan	3,205	3,143
Residential whole loans held at fair value excluded, purchase transaction not occurred	$ 92,800