Loans Held for Investment and Loans Held for Investment at Fair Value - Schedule of Nonaccrual With No Allowance for Loan Loss and Total Nonaccrual of Loans Held for Investment (Details) - USD ($) $ in Thousands	Jun. 30, 2022	Dec. 31, 2021	Jun. 30, 2021
Financing Receivable Impaired [Line Items]
Total Nonaccrual	$ 255,202	$ 276,418
Nonaccrual with No Allowance for Loan Loss	243,430	263,026
Nonaccrual with Allowance for Loan Loss	11,772	13,392
Allowance for Loans Individually Evaluated	$ 1,164	$ 1,406	$ 1,949
Percentage of Allowance to Total Nonaccrual Loans with Allowance	9.90%	10.50%
Troubled Debt Restructuring included in nonaccrual loans:	$ 160	$ 165
Commercial - Purchase
Financing Receivable Impaired [Line Items]
Total Nonaccrual	14,911	17,260
Nonaccrual with No Allowance for Loan Loss	14,647	16,501
Nonaccrual with Allowance for Loan Loss	264	759
Allowance for Loans Individually Evaluated	$ 6	$ 9	57
Percentage of Allowance to Total Nonaccrual Loans with Allowance	0.10%	0.10%
Commercial - Refinance
Financing Receivable Impaired [Line Items]
Total Nonaccrual	$ 80,442	$ 85,935
Nonaccrual with No Allowance for Loan Loss	76,077	79,131
Nonaccrual with Allowance for Loan Loss	4,365	6,804
Allowance for Loans Individually Evaluated	$ 434	$ 826	1,005
Percentage of Allowance to Total Nonaccrual Loans with Allowance	3.70%	6.20%
Residential 1-4 Unit - Purchase
Financing Receivable Impaired [Line Items]
Total Nonaccrual	$ 22,885	$ 17,385
Nonaccrual with No Allowance for Loan Loss	22,542	17,128
Nonaccrual with Allowance for Loan Loss	343	257
Allowance for Loans Individually Evaluated	$ 52	$ 96	120
Percentage of Allowance to Total Nonaccrual Loans with Allowance	0.40%	0.70%
Residential 1-4 Unit - Refinance
Financing Receivable Impaired [Line Items]
Total Nonaccrual	$ 96,649	$ 107,552
Nonaccrual with No Allowance for Loan Loss	94,594	105,515
Nonaccrual with Allowance for Loan Loss	2,055	2,037
Allowance for Loans Individually Evaluated	$ 196	$ 138	342
Percentage of Allowance to Total Nonaccrual Loans with Allowance	1.70%	1.00%
Short Term 1-4 Unit - Purchase
Financing Receivable Impaired [Line Items]
Total Nonaccrual	$ 2,302	$ 2,986
Nonaccrual with No Allowance for Loan Loss	2,302	2,881
Nonaccrual with Allowance for Loan Loss		105
Allowance for Loans Individually Evaluated		$ 31	56
Percentage of Allowance to Total Nonaccrual Loans with Allowance		0.20%
Short Term 1-4 Unit - Refinance
Financing Receivable Impaired [Line Items]
Total Nonaccrual	38,013	$ 45,300
Nonaccrual with No Allowance for Loan Loss	33,268	41,870
Nonaccrual with Allowance for Loan Loss	4,745	3,430
Allowance for Loans Individually Evaluated	$ 476	$ 306	$ 369
Percentage of Allowance to Total Nonaccrual Loans with Allowance	4.00%	2.30%
Individual Loan Evaluation
Financing Receivable Impaired [Line Items]
Troubled Debt Restructuring included in nonaccrual loans:	$ 25	$ 25