Loans Held for Investment and Loans Held for Investment at Fair Value - Schedule of Nonaccrual With No Allowance for Loan Loss and Total Nonaccrual of Loans Held for Investment (Details) - USD ($) $ in Thousands	Mar. 31, 2023	Dec. 31, 2022	Mar. 31, 2022
Financing Receivable Impaired [Line Items]
Total Nonaccrual	$ 309,714	$ 295,339
Nonaccrual with No Allowance for Loan Loss	297,186	282,777
Nonaccrual with Allowance for Loan Loss	12,528	12,562
Allowance for Loans Individually Evaluated	$ 996	$ 1,096	$ 1,599
Percentage of Allowance to Total Nonaccrual Loans with Allowance	7.90%	8.70%
Commercial - Purchase
Financing Receivable Impaired [Line Items]
Total Nonaccrual	$ 19,422	$ 22,571
Nonaccrual with No Allowance for Loan Loss	18,486	22,437
Nonaccrual with Allowance for Loan Loss	936	134
Allowance for Loans Individually Evaluated	$ 138	$ 28	28
Percentage of Allowance to Total Nonaccrual Loans with Allowance	1.10%	0.20%
Commercial - Refinance
Financing Receivable Impaired [Line Items]
Total Nonaccrual	$ 87,439	$ 87,133
Nonaccrual with No Allowance for Loan Loss	83,418	82,330
Nonaccrual with Allowance for Loan Loss	4,021	4,803
Allowance for Loans Individually Evaluated	$ 416	$ 517	770
Percentage of Allowance to Total Nonaccrual Loans with Allowance	3.30%	4.10%
Residential 1-4 Unit - Purchase
Financing Receivable Impaired [Line Items]
Total Nonaccrual	$ 34,171	$ 27,984
Nonaccrual with No Allowance for Loan Loss	34,171	27,516
Nonaccrual with Allowance for Loan Loss		468
Allowance for Loans Individually Evaluated		$ 118	96
Percentage of Allowance to Total Nonaccrual Loans with Allowance		0.90%
Residential 1-4 Unit - Refinance
Financing Receivable Impaired [Line Items]
Total Nonaccrual	124,390	$ 113,909
Nonaccrual with No Allowance for Loan Loss	120,732	111,742
Nonaccrual with Allowance for Loan Loss	3,659	2,167
Allowance for Loans Individually Evaluated	$ 204	$ 175	198
Percentage of Allowance to Total Nonaccrual Loans with Allowance	1.60%	1.40%
Short Term 1-4 Unit - Purchase
Financing Receivable Impaired [Line Items]
Total Nonaccrual	$ 8,447	$ 8,140
Nonaccrual with No Allowance for Loan Loss	8,447	8,140
Allowance for Loans Individually Evaluated			32
Short Term 1-4 Unit - Refinance
Financing Receivable Impaired [Line Items]
Total Nonaccrual	35,845	35,602
Nonaccrual with No Allowance for Loan Loss	31,932	30,612
Nonaccrual with Allowance for Loan Loss	3,912	4,990
Allowance for Loans Individually Evaluated	$ 238	$ 258	$ 475
Percentage of Allowance to Total Nonaccrual Loans with Allowance	1.90%	2.10%
Individual Loan Evaluation
Financing Receivable Impaired [Line Items]
Troubled Debt Restructuring included in nonaccrual loans:		$ 25