Fair Value Measurements - Narrative (Details) $ in Thousands	3 Months Ended		9 Months Ended	12 Months Ended
	Jun. 30, 2020 USD ($)	Mar. 31, 2020 USD ($)	Sep. 30, 2021 USD ($) loan	Dec. 31, 2020 USD ($) loan
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Securitized debt at fair value, collateralized by Loans held for investment			$ 7,947,644	$ 8,711,677
Level 3 | Jumbo, Prime Residential Mortgages
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Securitized debt at fair value, collateralized by Loans held for investment			$ 40,000
Investments with Difference Between Model Price and Third-Party Price | Mortgage Backed Securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Loans internally valued lower than third party prices | loan			25	23
Excess (deficit) of third party prices over internally developed price, assets			$ 35,000	$ (3,000)
Investments with Difference Between Model Price and Third-Party Price | Securitized Loans Held For Investment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Excess (deficit) of third party prices over internally developed price, assets			74,000	55,000
Internal Assessment | Investments with Difference Between Model Price and Third-Party Price | Mortgage Backed Securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets, fair value disclosure			$ 383,000	$ 389,000
Internal Assessment | Investments with Difference Between Model Price and Third-Party Price | Securitized Loans Held For Investment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Loans internally valued lower than third party prices | loan			3	3
Assets, fair value disclosure			$ 3,800,000	$ 503,000
Third Party Assessment | Investments with Difference Between Model Price and Third-Party Price | Mortgage Backed Securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Assets, fair value disclosure			348,000	392,000
Third Party Assessment | Investments with Difference Between Model Price and Third-Party Price | Securitized Loans Held For Investment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Excess (deficit) of third party prices over internally developed price, assets			3,700,000	448,000
Non-Agency RMBS
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Transfers out of level 3	$ 135,000	$ 135,000
Non-Agency RMBS | Level 3
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Transfers out of level 3			0	135,118
Loans Held for Investment at Fair Value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Transfers out of level 3			272,000
Loans Held for Investment at Fair Value | Level 3
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Transfers out of level 3			$ 272,198
Securitized debt | Non-Agency RMBS | Internal Assessment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Loans internally valued lower than third party prices | loan			2
Liabilities, fair value disclosure				209,000
Excess (deficit) of third party prices over internally developed price, liabilities				17,000
Securitized debt | Non-Agency RMBS | Third Party Assessment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Liabilities, fair value disclosure				$ 192,000