Fair Value of Financial Instruments - Difference Between Fair Value and Unpaid Principal Balance (Details) - USD ($) $ in Thousands	Dec. 31, 2016	Dec. 31, 2015
Fair Value, Option, Quantitative Disclosures [Line Items]
Total loans, Fair Value	$ 814,682	$ 623,300
Total loans, Unpaid Principal Balance	966,276	786,330
Residential whole loans
Fair Value, Option, Quantitative Disclosures [Line Items]
Total loans, Fair Value	814,682	623,276
Total loans, Unpaid Principal Balance	966,174	786,330
Total loans, Difference	(151,492)	(163,054)
Loans 90 days or more past due, Fair Value	570,025	493,640
Loans 90 days or more past due, Unpaid Principal Balance	695,282	637,459
Loans 90 days or more past due, Difference	$ (125,257)	$ (143,819)