Fair Value Measurements - Narrative (Details) $ in Thousands	3 Months Ended
	Mar. 31, 2020 USD ($) loan Loan	Mar. 31, 2019 USD ($)	Dec. 31, 2019 USD ($) loan
Investments with Difference Between Model Price and Third-Party Price | Mortgage Backed Securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Loans internally valued lower than third party prices | loan	60		6
Excess (deficit) of third party prices over internally developed price, assets	$ 13,000		$ 3,000
Investments with Difference Between Model Price and Third-Party Price | Securitized Loans Held For Investment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Excess (deficit) of third party prices over internally developed price, assets	56,000		20,000
Internal Assessment | Investments with Difference Between Model Price and Third-Party Price | Mortgage Backed Securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets, fair value disclosure	893,000		22,000
Internal Assessment | Investments with Difference Between Model Price and Third-Party Price | Securitized Loans Held For Investment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets, fair value disclosure	$ 532,000		$ 147,000
Number of loans | loan	5		1
Third Party Assessment | Investments with Difference Between Model Price and Third-Party Price | Mortgage Backed Securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets, fair value disclosure	$ 880,000		$ 19,000
Third Party Assessment | Investments with Difference Between Model Price and Third-Party Price | Securitized Loans Held For Investment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Excess (deficit) of third party prices over internally developed price, assets	476,000		$ 127,000
Non-Agency RMBS
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Transfers out of level 3	$ 135,118	$ 0
Securitized debt | Internal Assessment | Investments with Difference Between Model Price and Third-Party Price
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Loans internally valued lower than third party prices | Loan	8
Liabilities, fair value disclosure	$ 148,000
Excess (deficit) of third party prices over internally developed price, liabilities	45,000
Securitized debt | Third Party Assessment | Investments with Difference Between Model Price and Third-Party Price
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Liabilities, fair value disclosure	$ 103,000