Mortgage-Backed Debt - Schedule of Mortgage-Backed Debt (Details) - USD ($) $ in Thousands	Dec. 31, 2016	Dec. 31, 2015
Debt Instrument [Line Items]
Mortgage-backed debt at amortized cost	$ 429,931	$ 469,339
Mortgage-backed debt at fair value	514,025	582,340
Total mortgage-backed debt	$ 943,956	$ 1,051,679
Mortgage-Backed Debt [Member]
Debt Instrument [Line Items]
Weighted-Average Stated Interest Rate	5.87%	5.77%
Mortgage-Backed Debt [Member] | Residual Trusts [Member]
Debt Instrument [Line Items]
Unpaid principal balance	$ 434,667	$ 474,759
Weighted-Average Stated Interest Rate	6.07%	6.07%
Mortgage-Backed Debt [Member] | Non-Residual Trusts [Member]
Debt Instrument [Line Items]
Unpaid principal balance	$ 518,317	$ 585,839
Weighted-Average Stated Interest Rate	5.70%	5.53%