Investment Securities - Summary (Details) - USD ($) $ in Thousands	Jun. 30, 2022	Dec. 31, 2021
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreement to Repurchase	$ (128,365)	$ (609,251)
Senior
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	975	3,076
Allocated Capital	(74)	(1,013)
Mezzanine
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	2,145	2,178
Allocated Capital	549	547
Subordinate
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	55,414	80,058
Allocated Capital	42,799	80,058
Interest Only/Excess
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	10,133	15,052
Allocated Capital	10,133	15,052
Total AOMT RMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	68,667	100,364
Allocated Capital	53,407	94,644
Subordinate
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	3,354	10,292
Allocated Capital	3,354	10,292
Interest Only/Excess
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	2,634	2,923
Allocated Capital	2,634	2,923
Total Other Non-Agency RMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	5,988	13,215
Allocated Capital	5,988	13,215
Fannie Mae
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	820,101	281,225
Allocated Capital	706,996	13,939
Freddie Mac
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	28,103	90,830
Allocated Capital	28,103	3,335
Whole Pool Total Agency RMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	848,204	372,055
Allocated Capital	735,099	17,274
RMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	922,859	485,634
Securities Sold Under Agreement to Repurchase	(128,365)	(360,501)
Allocated Capital	794,494	125,133
Subordinate
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	6,573	7,993
Allocated Capital	6,573	7,993
Interest Only/Excess
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	2,409	2,763
Allocated Capital	2,409	2,763
Total AOMT CMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	8,982	10,756
Allocated Capital	8,982	10,756
Senior
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreement to Repurchase	(1,049)	(4,089)
Mezzanine
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreement to Repurchase	(1,596)	(1,631)
Subordinate
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreement to Repurchase	(12,615)	0
Interest Only/Excess
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreement to Repurchase	0	0
Total AOMT RMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreement to Repurchase	(15,260)	(5,720)
Subordinate
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreement to Repurchase	0	0
Interest Only/Excess
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreement to Repurchase	0	0
Total Other Non-Agency RMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreement to Repurchase	0	0
Fannie Mae
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreement to Repurchase	(113,105)	(267,286)
Freddie Mac
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreement to Repurchase	0	(87,495)
Whole Pool Total Agency RMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreement to Repurchase	(113,105)	(354,781)
RMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreement to Repurchase	(128,365)	(360,501)
Subordinate
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreement to Repurchase	0	0
Interest Only/Excess
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreement to Repurchase	0	0
Total AOMT CMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreement to Repurchase	$ 0	$ 0