Mortgage-Backed Debt - Schedule of Mortgage-Backed Debt (Details) - USD ($) $ in Thousands	Dec. 31, 2018	Dec. 31, 2017
Debt Instrument [Line Items]
Mortgage-backed debt at fair value	$ 131,313	$ 348,682
Mortgage-backed debt at amortized cost	0	387,200
Total mortgage-backed debt	$ 131,313	$ 735,882
Mortgage Backed Debt At Fair Value [Member]
Debt Instrument [Line Items]
Weighted-average stated interest rate	6.56%	6.26%
Mortgage Backed Debt At Amortized Cost [Member]
Debt Instrument [Line Items]
Weighted-average stated interest rate	0.00%	6.07%
Mortgage-Backed Debt [Member]
Debt Instrument [Line Items]
Weighted-average stated interest rate	6.56%	6.16%
Non-Residual Trusts [Member] | Mortgage-Backed Debt [Member]
Debt Instrument [Line Items]
Unpaid principal balance	$ 139,064	$ 353,262
Residual Trusts [Member] | Mortgage Backed Debt At Amortized Cost [Member]
Debt Instrument [Line Items]
Unpaid principal balance		$ 391,208