Investment Securities - Narrative (Details)	12 Months Ended				132 Months Ended
	Dec. 31, 2023 USD ($) security shares	Dec. 31, 2022 USD ($) security	Dec. 31, 2021 USD ($)	Dec. 31, 2012 shares	Dec. 31, 2022 USD ($) security shares	Dec. 31, 2020 USD ($)
Debt Securities, Available-for-sale [Line Items]
Cost of third party management	$ 500,000	$ 600,000	$ 900,000
Number of securities with unrealized losses | security	14	10			10
Credit loss allowance	$ 13,143,000	$ 3,182,000	8,610,000		$ 3,182,000	$ 5,675,000
Amortized Cost Basis	$ 619,397,000	676,652,000			$ 676,652,000
SEREF
Debt Securities, Available-for-sale [Line Items]
Number of shares acquired (in shares) | shares				9,140,000
Number of shares redeemed (in shares) | shares	1,892,313				0
Proceeds from shares redeemed	$ 2,500,000
Remaining held (in shares) | shares	7,247,687
Equity security, fair value	$ 8,300,000	9,800,000			$ 9,800,000
Ownership percentage	2.30%
Fair value option
Debt Securities, Available-for-sale [Line Items]
Fair value of mortgage backed securities	$ 1,600,000,000
VIE eliminations
Debt Securities, Available-for-sale [Line Items]
Fair value of investment securities before consolidation of VIEs eliminated against VIE liabilities	18,600,000
RMBS
Debt Securities, Available-for-sale [Line Items]
Portion of securities with variable rate	$ 91,300,000
RMBS | Available-for-sale
Debt Securities, Available-for-sale [Line Items]
Portion of securities with variable rate (as a percent)	89.00%
RMBS | Fair value option
Debt Securities, Available-for-sale [Line Items]
Portion of securities with variable rate	$ 0
Fair value of investment securities before consolidation of VIEs	449,900,000
Unpaid principal balance of investment securities before consolidation of VIEs	326,300,000
CMBS
Debt Securities, Available-for-sale [Line Items]
Credit loss allowance	164,000	172,000	3,140,000		172,000	0
Amortized Cost Basis	580,704,000	577,681,000			577,681,000
CMBS | Fair value option
Debt Securities, Available-for-sale [Line Items]
Portion of securities with variable rate	0
Fair value of investment securities before consolidation of VIEs	1,100,000,000
Unpaid principal balance of investment securities before consolidation of VIEs	2,700,000,000
Infrastructure bonds
Debt Securities, Available-for-sale [Line Items]
Credit loss allowance	10,081,000	3,010,000	$ 2,908,000		3,010,000	$ 2,926,000
Amortized Cost Basis	29,123,000	$ 69,214,000			$ 69,214,000
Infrastructure bonds | Credit deteriorated | Infrastructure Portfolio Segment | Choctaw, Mississippi
Debt Securities, Available-for-sale [Line Items]
Credit loss allowance	7,200,000
Allowance for credit loss to date	10,000,000
Amortized Cost Basis	$ 19,200,000