Investment Securities - Narrative (Details)	3 Months Ended		12 Months Ended
	Mar. 31, 2024 USD ($) security shares	Mar. 31, 2023 USD ($) shares	Dec. 31, 2012 shares	Dec. 31, 2023 USD ($) security
Debt Securities, Available-for-sale [Line Items]
Cost of third party management	$ 200,000	$ 200,000
Number of securities with unrealized losses | security	15			14
Amortized cost basis	$ 606,592,000			$ 619,397,000
SEREF
Debt Securities, Available-for-sale [Line Items]
Number of shares acquired (in shares) | shares			9,140,000
Number of shares redeemed (in shares) | shares	1,005,348	0
Proceeds from shares redeemed	$ 1,300,000
Remaining held (in shares) | shares	6,242,339
Equity security, fair value	$ 7,300,000			8,300,000
Ownership percentage	2.30%
VIE eliminations
Debt Securities, Available-for-sale [Line Items]
Fair value of investment securities before consolidation of VIEs eliminated against VIE liabilities	$ 19,500,000
Fair Value Option
Debt Securities, Available-for-sale [Line Items]
Fair value of mortgage backed securities	1,600,000,000
RMBS
Debt Securities, Available-for-sale [Line Items]
Portion of securities with variable rate	$ 89,400,000
RMBS | Available-for-sale
Debt Securities, Available-for-sale [Line Items]
Portion of securities with variable rate (as a percent)	89.00%
RMBS | Fair Value Option
Debt Securities, Available-for-sale [Line Items]
Portion of securities with variable rate	$ 0
Fair value of investment securities before consolidation of VIEs	434,900,000
Unpaid principal balance of investment securities before consolidation of VIEs	326,300,000
CMBS
Debt Securities, Available-for-sale [Line Items]
Amortized cost basis	561,174,000			580,704,000
CMBS | Fair Value Option
Debt Securities, Available-for-sale [Line Items]
Portion of securities with variable rate	0
Fair value of investment securities before consolidation of VIEs	1,100,000,000
Unpaid principal balance of investment securities before consolidation of VIEs	2,700,000,000
Infrastructure bonds
Debt Securities, Available-for-sale [Line Items]
Amortized cost basis	28,502,000			29,123,000
Infrastructure bonds | Choctaw, Mississippi | Infrastructure Portfolio Segment | Credit deteriorated
Debt Securities, Available-for-sale [Line Items]
Credit loss allowance	10,000,000
Amortized cost basis	19,200,000
Preferred interests
Debt Securities, Available-for-sale [Line Items]
Amortized cost basis	16,916,000			$ 9,570,000
Preferred interests | California
Debt Securities, Available-for-sale [Line Items]
Debt securities, held-to-maturity, nonaccrual	9,800,000
Preferred interests | Texas
Debt Securities, Available-for-sale [Line Items]
Debt securities, held-to-maturity, nonaccrual	$ 7,100,000