Fair Value Measurements - Carrying Value and Fair Value of Financial Instruments Not Carried at Fair Value on a Recurring Basis (Details) - USD ($) $ in Thousands	Sep. 30, 2021	Dec. 31, 2020
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securitized debt, collateralized by Non-Agency RMBS	$ 92,204	$ 113,433
Long term debt	0	51,623
Carrying Amount | Level 2
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Secured financing agreements	3,788,336	4,636,847
Long term debt		51,623
Carrying Amount | Level 3
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securitized debt, collateralized by Non-Agency RMBS	92,204	113,433
Fair Value | Level 2
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Secured financing agreements	3,791,783	4,803,256
Long term debt		80,750
Fair Value | Level 3
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securitized debt, collateralized by Non-Agency RMBS	$ 73,633	$ 97,097