Fair Value Measurements - Narrative (Details) $ in Thousands	3 Months Ended	6 Months Ended	12 Months Ended
	Mar. 31, 2020 USD ($)	Jun. 30, 2020 USD ($) loan	Dec. 31, 2019 USD ($) loan
Investments with Difference Between Model Price and Third-Party Price | Mortgage Backed Securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Loans internally valued lower than third party prices | loan		40	6
Excess (deficit) of third party prices over internally developed price, assets		$ (58,000)	$ 3,000
Investments with Difference Between Model Price and Third-Party Price | Securitized Loans Held For Investment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Excess (deficit) of third party prices over internally developed price, assets		12,000	20,000
Internal Assessment | Investments with Difference Between Model Price and Third-Party Price | Mortgage Backed Securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets, fair value disclosure		1,100,000	22,000
Internal Assessment | Investments with Difference Between Model Price and Third-Party Price | Securitized Loans Held For Investment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets, fair value disclosure		$ 64,000	$ 147,000
Number of loans | loan		3	1
Third Party Assessment | Investments with Difference Between Model Price and Third-Party Price | Mortgage Backed Securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets, fair value disclosure		$ 1,200,000	$ 19,000
Third Party Assessment | Investments with Difference Between Model Price and Third-Party Price | Securitized Loans Held For Investment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Excess (deficit) of third party prices over internally developed price, assets		52,000	127,000
Non-Agency RMBS
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Transfers out of level 3	$ 135,000	$ 135,118	$ 0
Securitized debt | Internal Assessment | Investments with Difference Between Model Price and Third-Party Price
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Loans internally valued lower than third party prices | loan		3
Liabilities, fair value disclosure		$ 240,000
Excess (deficit) of third party prices over internally developed price, liabilities		(22,000)
Securitized debt | Third Party Assessment | Investments with Difference Between Model Price and Third-Party Price
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Liabilities, fair value disclosure		$ 262,000