Loans Held for Investment and Loans Held for Investment at Fair Value - Schedule of Accrued Interest Receivables Written Off by Reversing Interest Income by Portfolio Segment (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2024	Jun. 30, 2023	Jun. 30, 2024	Jun. 30, 2023
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	$ 2,624,859	$ 3,062,566	$ 2,624,859	$ 3,062,566
Financing Receivable, Accrued Interest, Writeoff	1,545	2,514	2,856	4,295
Commercial - Purchase
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	599,314	664,482	599,314	664,482
Financing Receivable, Accrued Interest, Writeoff	147	136	202	268
Commercial - Refinance
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	758,179	854,048	758,179	854,048
Financing Receivable, Accrued Interest, Writeoff	762	667	1,629	1,183
Residential 1-4 Unit - Purchase
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	469,400	553,269	469,400	553,269
Financing Receivable, Accrued Interest, Writeoff	151	476	286	773
Residential 1-4 Unit - Refinance
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	734,003	882,020	734,003	882,020
Financing Receivable, Accrued Interest, Writeoff	421	881	614	1,597
Short Term 1-4 Unit - Purchase
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	33,113	49,242	33,113	49,242
Financing Receivable, Accrued Interest, Writeoff	10	7	10	31
Short Term 1-4 Unit - Refinance
Financing Receivable Impaired [Line Items]
Financing Receivable, Amortized Cost Basis	30,850	59,505	30,850	59,505
Financing Receivable, Accrued Interest, Writeoff	$ 54	$ 347	$ 115	$ 443