Loans Held for Investment and Loans Held for Investment at Fair Value - Schedule of Nonaccrual With No Allowance for Loan Loss and Total Nonaccrual of Loans Held for Investment (Details) - USD ($) $ in Thousands	Dec. 31, 2023	Dec. 31, 2022
Financing Receivable Impaired [Line Items]
Total Nonaccrual	$ 325,354	$ 295,339
Nonaccrual with No Allowance for Loan Loss	315,906	282,777
Nonaccrual with Allowance for Loan Loss	9,449	12,562
Allowance for Loans Individually Evaluated	974	1,096
Troubled debt restructuring included in nonaccrual loans:		165
Commercial - Purchase
Financing Receivable Impaired [Line Items]
Total Nonaccrual	28,221	22,571
Nonaccrual with No Allowance for Loan Loss	27,037	22,437
Nonaccrual with Allowance for Loan Loss	1,184	134
Allowance for Loans Individually Evaluated	156	28
Commercial - Refinance
Financing Receivable Impaired [Line Items]
Total Nonaccrual	86,890	87,133
Nonaccrual with No Allowance for Loan Loss	84,575	82,330
Nonaccrual with Allowance for Loan Loss	2,315	4,803
Allowance for Loans Individually Evaluated	444	517
Residential 1-4 Unit - Purchase
Financing Receivable Impaired [Line Items]
Total Nonaccrual	36,253	27,984
Nonaccrual with No Allowance for Loan Loss	36,253	27,516
Nonaccrual with Allowance for Loan Loss		468
Allowance for Loans Individually Evaluated		118
Residential 1-4 Unit - Refinance
Financing Receivable Impaired [Line Items]
Total Nonaccrual	137,925	113,909
Nonaccrual with No Allowance for Loan Loss	134,579	111,742
Nonaccrual with Allowance for Loan Loss	3,346	2,167
Allowance for Loans Individually Evaluated	245	175
Short Term 1-4 Unit - Purchase
Financing Receivable Impaired [Line Items]
Total Nonaccrual	6,402	8,140
Nonaccrual with No Allowance for Loan Loss	6,403	8,140
Short Term 1-4 Unit - Refinance
Financing Receivable Impaired [Line Items]
Total Nonaccrual	29,663	35,602
Nonaccrual with No Allowance for Loan Loss	27,059	30,612
Nonaccrual with Allowance for Loan Loss	2,604	4,990
Allowance for Loans Individually Evaluated	$ 129	258
Individual Loan Evaluation
Financing Receivable Impaired [Line Items]
Troubled debt restructuring included in nonaccrual loans:		$ 25