Investment Securities - Narrative (Details)	3 Months Ended		9 Months Ended		12 Months Ended
	Sep. 30, 2023 USD ($) security shares	Sep. 30, 2022 USD ($)	Sep. 30, 2023 USD ($) security shares	Sep. 30, 2022 USD ($)	Dec. 31, 2012 shares	Dec. 31, 2022 USD ($) security
Debt Securities, Available-for-sale [Line Items]
Cost of third party management	$ 200,000	$ 200,000	$ 600,000	$ 800,000
Number of securities with unrealized losses | security	14		14			10
Credit loss allowance	$ 12,994,000		$ 12,994,000			$ 3,182,000
Amortized Cost Basis	$ 604,942,000		$ 604,942,000			676,652,000
SEREF
Debt Securities, Available-for-sale [Line Items]
Number of shares acquired (in shares) | shares					9,140,000
Number of shares redeemed (in shares) | shares	672,166		895,182
Proceeds from shares redeemed	$ 900,000		$ 1,200,000
Remaining held (in shares) | shares	8,244,818		8,244,818
Equity security, fair value	$ 8,800,000		$ 8,800,000			9,800,000
Ownership percentage			2.00%
Fair Value Option
Debt Securities, Available-for-sale [Line Items]
Fair value of mortgage backed securities	1,700,000,000		$ 1,700,000,000
VIE eliminations
Debt Securities, Available-for-sale [Line Items]
Fair value of investment securities before consolidation of VIEs eliminated against VIE liabilities	18,700,000		18,700,000
RMBS
Debt Securities, Available-for-sale [Line Items]
Portion of securities with variable rate	$ 91,100,000		$ 91,100,000
RMBS | Available-for-sale
Debt Securities, Available-for-sale [Line Items]
Portion of securities with variable rate (as a percent)	89.00%		89.00%
RMBS | Fair Value Option
Debt Securities, Available-for-sale [Line Items]
Portion of securities with variable rate	$ 0		$ 0
Fair value of investment securities before consolidation of VIEs	451,200,000		451,200,000
Unpaid principal balance of investment securities before consolidation of VIEs	326,300,000		326,300,000
CMBS
Debt Securities, Available-for-sale [Line Items]
Credit loss allowance	492,000		492,000			172,000
Amortized Cost Basis	569,867,000		569,867,000			577,681,000
CMBS | Fair Value Option
Debt Securities, Available-for-sale [Line Items]
Portion of securities with variable rate	102,900,000		102,900,000
Fair value of investment securities before consolidation of VIEs	1,200,000,000		1,200,000,000
Unpaid principal balance of investment securities before consolidation of VIEs	2,800,000,000		2,800,000,000
Infrastructure bonds
Debt Securities, Available-for-sale [Line Items]
Credit loss allowance	10,082,000		10,082,000			3,010,000
Amortized Cost Basis	29,664,000		29,664,000			$ 69,214,000
Infrastructure bonds | Credit deteriorated | Infrastructure Portfolio Segment | Choctaw, Mississippi
Debt Securities, Available-for-sale [Line Items]
Credit loss allowance	7,200,000		7,200,000
Allowance for credit loss to date	10,000,000		10,000,000
Amortized Cost Basis	$ 19,200,000		$ 19,200,000