Loans Held for Investment and Loans Held for Investment at Fair Value - Schedule of Aging Status of Amortized Cost Basis in Loans Held for Investment Portfolio (Details) - COVID19 Pandemic - Aging Status and Accrual Status - USD ($) $ in Thousands	Jun. 30, 2024	Dec. 31, 2023
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	$ 2,624,859	$ 2,832,892
Loans Individually Evaluated
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	327,290	325,354
Loans Individually Evaluated | Commercial - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	30,198	28,221
Loans Individually Evaluated | Commercial - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	99,844	86,890
Loans Individually Evaluated | Residential 1-4 Unit - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	33,640	36,253
Loans Individually Evaluated | Residential 1-4 Unit - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	132,448	137,925
Loans Individually Evaluated | Short Term 1-4 Unit - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	6,904	6,402
Loans Individually Evaluated | Short Term 1-4 Unit - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	24,256	29,663
Loans Collectively Evaluated
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	2,297,569	2,507,538
Loans Collectively Evaluated | Commercial - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	569,116	603,704
Loans Collectively Evaluated | Commercial - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	658,335	712,926
Loans Collectively Evaluated | Residential 1-4 Unit - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	435,760	474,827
Loans Collectively Evaluated | Residential 1-4 Unit - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	601,555	670,224
Loans Collectively Evaluated | Short Term 1-4 Unit - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	26,209	34,709
Loans Collectively Evaluated | Short Term 1-4 Unit - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	6,594	11,148
30-59 Days Past Due
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	155,065	165,297
30-59 Days Past Due | Loans Individually Evaluated
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	8,393	10,632
30-59 Days Past Due | Loans Individually Evaluated | Commercial - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	531	2,329
30-59 Days Past Due | Loans Individually Evaluated | Commercial - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	3,931	4,716
30-59 Days Past Due | Loans Individually Evaluated | Residential 1-4 Unit - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	252	544
30-59 Days Past Due | Loans Individually Evaluated | Residential 1-4 Unit - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	3,679	2,988
30-59 Days Past Due | Loans Individually Evaluated | Short Term 1-4 Unit - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans		55
30-59 Days Past Due | Loans Collectively Evaluated
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	146,672	154,665
30-59 Days Past Due | Loans Collectively Evaluated | Commercial - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	24,496	21,342
30-59 Days Past Due | Loans Collectively Evaluated | Commercial - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	36,638	47,430
30-59 Days Past Due | Loans Collectively Evaluated | Residential 1-4 Unit - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	28,494	29,236
30-59 Days Past Due | Loans Collectively Evaluated | Residential 1-4 Unit - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	50,444	52,510
30-59 Days Past Due | Loans Collectively Evaluated | Short Term 1-4 Unit - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	6,152	1,169
30-59 Days Past Due | Loans Collectively Evaluated | Short Term 1-4 Unit - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	448	2,978
60-89 Days Past Due
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	53,669	55,899
60-89 Days Past Due | Loans Individually Evaluated
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	3,789	4,996
60-89 Days Past Due | Loans Individually Evaluated | Commercial - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	920	668
60-89 Days Past Due | Loans Individually Evaluated | Commercial - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	1,290	2,405
60-89 Days Past Due | Loans Individually Evaluated | Residential 1-4 Unit - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	574
60-89 Days Past Due | Loans Individually Evaluated | Residential 1-4 Unit - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	1,005	1,923
60-89 Days Past Due | Loans Collectively Evaluated
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	49,880	50,903
60-89 Days Past Due | Loans Collectively Evaluated | Commercial - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	7,297	8,352
60-89 Days Past Due | Loans Collectively Evaluated | Commercial - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	14,099	14,002
60-89 Days Past Due | Loans Collectively Evaluated | Residential 1-4 Unit - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	5,600	6,850
60-89 Days Past Due | Loans Collectively Evaluated | Residential 1-4 Unit - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	22,453	20,828
60-89 Days Past Due | Loans Collectively Evaluated | Short Term 1-4 Unit - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	170	658
60-89 Days Past Due | Loans Collectively Evaluated | Short Term 1-4 Unit - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	261	213
90+Days Past Due
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	315,108	309,726
90+Days Past Due | Loans Individually Evaluated
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	315,108	309,726
90+Days Past Due | Loans Individually Evaluated | Commercial - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	28,747	25,224
90+Days Past Due | Loans Individually Evaluated | Commercial - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	94,623	79,769
90+Days Past Due | Loans Individually Evaluated | Residential 1-4 Unit - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	32,814	35,709
90+Days Past Due | Loans Individually Evaluated | Residential 1-4 Unit - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	127,764	133,014
90+Days Past Due | Loans Individually Evaluated | Short Term 1-4 Unit - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	6,904	6,402
90+Days Past Due | Loans Individually Evaluated | Short Term 1-4 Unit - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	24,256	29,608
Past Due
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	523,842	530,922
Past Due | Loans Individually Evaluated
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	327,290	325,354
Past Due | Loans Individually Evaluated | Commercial - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	30,198	28,221
Past Due | Loans Individually Evaluated | Commercial - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	99,844	86,890
Past Due | Loans Individually Evaluated | Residential 1-4 Unit - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	33,640	36,253
Past Due | Loans Individually Evaluated | Residential 1-4 Unit - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	132,448	137,925
Past Due | Loans Individually Evaluated | Short Term 1-4 Unit - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	6,904	6,402
Past Due | Loans Individually Evaluated | Short Term 1-4 Unit - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	24,256	29,663
Past Due | Loans Collectively Evaluated
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	196,552	205,568
Past Due | Loans Collectively Evaluated | Commercial - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	31,793	29,694
Past Due | Loans Collectively Evaluated | Commercial - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	50,737	61,432
Past Due | Loans Collectively Evaluated | Residential 1-4 Unit - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	34,094	36,086
Past Due | Loans Collectively Evaluated | Residential 1-4 Unit - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	72,897	73,338
Past Due | Loans Collectively Evaluated | Short Term 1-4 Unit - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	6,322	1,827
Past Due | Loans Collectively Evaluated | Short Term 1-4 Unit - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	709	3,191
Current
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	2,101,017	2,301,970
Current | Loans Collectively Evaluated
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	2,101,017	2,301,970
Current | Loans Collectively Evaluated | Commercial - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	537,323	574,010
Current | Loans Collectively Evaluated | Commercial - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	607,598	651,494
Current | Loans Collectively Evaluated | Residential 1-4 Unit - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	401,666	438,741
Current | Loans Collectively Evaluated | Residential 1-4 Unit - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	528,658	596,886
Current | Loans Collectively Evaluated | Short Term 1-4 Unit - Purchase
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	19,887	32,882
Current | Loans Collectively Evaluated | Short Term 1-4 Unit - Refinance
Financing Receivable Recorded Investment Past Due [Line Items]
Total loans	$ 5,885	$ 7,957