Fair Value Measurements - Financial Assets and Liabilities Carried at Fair Value on a Recurring Basis (Details) - USD ($) $ in Thousands	Sep. 30, 2021	Dec. 31, 2020
Assets:
Non-Agency RMBS, at fair value	$ 1,890,030	$ 2,150,714
Agency RMBS, at fair value	65,889	90,738
Agency CMBS, at fair value	1,062,131	1,740,368
Loans held for investment, at fair value	12,533,864	13,112,129
Liabilities:
Securitized debt at fair value, collateralized by Loans held for investment	7,947,644	8,711,677
Fair Value, Measurements, Recurring
Assets:
Non-Agency RMBS, at fair value	1,890,030	2,150,714
Agency RMBS, at fair value	65,889	90,738
Agency CMBS, at fair value	1,062,131
Loans held for investment, at fair value	12,533,864	13,112,129
Liabilities:
Securitized debt at fair value, collateralized by Loans held for investment	7,947,644	8,711,677
Fair Value, Measurements, Recurring | Level 1
Assets:
Non-Agency RMBS, at fair value	0	0
Agency RMBS, at fair value	0	0
Agency CMBS, at fair value	0	0
Loans held for investment, at fair value	0	0
Liabilities:
Securitized debt at fair value, collateralized by Loans held for investment	0	0
Fair Value, Measurements, Recurring | Level 2
Assets:
Non-Agency RMBS, at fair value	0	0
Agency RMBS, at fair value	65,889	90,738
Agency CMBS, at fair value	1,062,131	1,740,368
Loans held for investment, at fair value	307,004	0
Liabilities:
Securitized debt at fair value, collateralized by Loans held for investment	0	0
Fair Value, Measurements, Recurring | Level 3
Assets:
Non-Agency RMBS, at fair value	1,890,030	2,150,714
Agency RMBS, at fair value	0	0
Agency CMBS, at fair value	0	0
Loans held for investment, at fair value	12,226,860	13,112,129
Liabilities:
Securitized debt at fair value, collateralized by Loans held for investment	$ 7,947,644	$ 8,711,677