Investment Securities - Schedule of Investments in RMBS and CMBS at Fair Value (Details) - USD ($) $ in Thousands	Mar. 31, 2023	Dec. 31, 2022
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreements to Repurchase	$ (442,214)	$ (52,544)
Mezzanine
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	9,965	1,958
Allocated Capital	8,593	488
Subordinate
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	51,417	49,578
Allocated Capital	22,741	24,596
Interest Only/Excess
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	12,327	10,424
Allocated Capital	10,643	8,918
Retained RMBS in VIEs | VIE - Primary Beneficiary
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	0	0
Allocated Capital	(12,979)	(24,586)
Total AOMT RMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	73,709	61,960
Allocated Capital	28,998	9,416
Fannie Mae
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	253,818	501,458
Allocated Capital	253,818	501,458
Freddie Mac
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	195,360	491,920
Allocated Capital	195,360	491,920
Whole Pool Total Agency RMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	449,178	993,378
Allocated Capital	449,178	993,378
RMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	522,887	1,055,338
Securities Sold Under Agreements to Repurchase	(44,711)	(52,544)
Allocated Capital	478,176	1,002,794
Subordinate
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	2,942	2,901
Allocated Capital	2,942	2,901
Interest Only/Excess
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	3,538	3,210
Allocated Capital	3,538	3,210
Total AOMT CMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Real Estate Securities at Fair Value	6,480	6,111
Allocated Capital	6,480	6,111
Mezzanine
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreements to Repurchase	(1,372)	(1,470)
Subordinate
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreements to Repurchase	(28,676)	(24,982)
Interest Only/Excess
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreements to Repurchase	(1,684)	(1,506)
Retained RMBS in VIEs | VIE - Primary Beneficiary
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreements to Repurchase	(12,979)	(24,586)
Retained RMBS in VIEs | VIE - Primary Beneficiary | Eliminations
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreements to Repurchase	116,700	110,500
Total AOMT RMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreements to Repurchase	(44,711)	(52,544)
Fannie Mae
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreements to Repurchase	0	0
Freddie Mac
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreements to Repurchase	0	0
Whole Pool Total Agency RMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreements to Repurchase	0	0
RMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreements to Repurchase	(44,711)	(52,544)
Subordinate
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreements to Repurchase	0	0
Interest Only/Excess
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreements to Repurchase	0	0
Total AOMT CMBS
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities Sold Under Agreements to Repurchase	$ 0	$ 0