Loans Held for Investment and Loans Held for Investment at Fair Value - Schedule of Nonaccrual With No Allowance for Loan Loss and Total Nonaccrual of Loans Held for Investment (Details) - USD ($) $ in Thousands	Dec. 31, 2022	Dec. 31, 2021
Financing Receivable Impaired [Line Items]
Total Nonaccrual	$ 295,339	$ 276,418
Nonaccrual with No Allowance for Loan Loss	282,777	263,026
Nonaccrual with Allowance for Loan Loss	12,562	13,392
Allowance for Loans Individually Evaluated	$ 1,096	$ 1,406
Percentage of Allowance to Total Nonaccrual/ Impaired Loans	8.70%	10.50%
Troubled debt restructuring included in nonaccrual loans:		$ 165
Commercial - Purchase
Financing Receivable Impaired [Line Items]
Total Nonaccrual	$ 22,571	17,260
Nonaccrual with No Allowance for Loan Loss	22,437	16,501
Nonaccrual with Allowance for Loan Loss	134	759
Allowance for Loans Individually Evaluated	$ 28	$ 9
Percentage of Allowance to Total Nonaccrual/ Impaired Loans	0.20%	0.10%
Commercial - Refinance
Financing Receivable Impaired [Line Items]
Total Nonaccrual	$ 87,133	$ 85,935
Nonaccrual with No Allowance for Loan Loss	82,330	79,131
Nonaccrual with Allowance for Loan Loss	4,803	6,804
Allowance for Loans Individually Evaluated	$ 517	$ 826
Percentage of Allowance to Total Nonaccrual/ Impaired Loans	4.10%	6.20%
Residential 1-4 Unit - Purchase
Financing Receivable Impaired [Line Items]
Total Nonaccrual	$ 27,984	$ 17,385
Nonaccrual with No Allowance for Loan Loss	27,516	17,128
Nonaccrual with Allowance for Loan Loss	468	257
Allowance for Loans Individually Evaluated	$ 118	$ 96
Percentage of Allowance to Total Nonaccrual/ Impaired Loans	0.90%	0.70%
Residential 1-4 Unit - Refinance
Financing Receivable Impaired [Line Items]
Total Nonaccrual	$ 113,909	$ 107,552
Nonaccrual with No Allowance for Loan Loss	111,742	105,515
Nonaccrual with Allowance for Loan Loss	2,167	2,037
Allowance for Loans Individually Evaluated	$ 175	$ 138
Percentage of Allowance to Total Nonaccrual/ Impaired Loans	1.40%	1.00%
Short Term 1-4 Unit - Purchase
Financing Receivable Impaired [Line Items]
Total Nonaccrual	$ 8,140	$ 2,986
Nonaccrual with No Allowance for Loan Loss	8,140	2,881
Nonaccrual with Allowance for Loan Loss		105
Allowance for Loans Individually Evaluated		$ 31
Percentage of Allowance to Total Nonaccrual/ Impaired Loans		0.20%
Short Term 1-4 Unit - Refinance
Financing Receivable Impaired [Line Items]
Total Nonaccrual	35,602	$ 45,300
Nonaccrual with No Allowance for Loan Loss	30,612	41,870
Nonaccrual with Allowance for Loan Loss	4,990	3,430
Allowance for Loans Individually Evaluated	$ 258	$ 306
Percentage of Allowance to Total Nonaccrual/ Impaired Loans	2.10%	2.30%
Individual Loan Evaluation
Financing Receivable Impaired [Line Items]
Troubled debt restructuring included in nonaccrual loans:	$ 25	$ 25