Loans Held for Investment and Loans Held for Investment at Fair Value - Schedule of Amortized Cost in Loans Held for Investment based on Accrual Status and by Loan Origination Year (Details) - Accrual Status - USD ($) $ in Thousands	3 Months Ended	12 Months Ended
	Dec. 31, 2023	Dec. 31, 2023	Dec. 31, 2022
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2023			$ 1,243,841
Term Loans Amortized Cost Basis by Origination Year, 2022	$ 1,090,714	$ 1,090,714	1,033,695
Term Loans Amortized Cost Basis by Origination Year, 2021	918,313	918,313	168,977
Term Loans Amortized Cost Basis by Origination Year, 2020	148,867	148,867	358,100
Term Loans Amortized Cost Basis by Origination Year, 2019	288,999	288,999	231,024
Term Loans Amortized Cost Basis by Origination Year, Pre-2019	385,999	385,999	241,647
Total	2,832,892	2,832,892	3,277,284
Term Loans Amortized Cost Basis by Origination Year, 2022, Gross Charge-offs	744	1,120
Term Loans Amortized Cost Basis by Origination Year, 2021, Gross Charge-offs		473
Term Loans Amortized Cost Basis by Origination Year, 2019, Gross Charge-offs		446
Loans and Lease Receivable Accrual Gross Writeoffs, Total	744	2,039
Commercial - Purchase
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2023			275,224
Term Loans Amortized Cost Basis by Origination Year, 2022	257,753	257,753	256,059
Term Loans Amortized Cost Basis by Origination Year, 2021	232,571	232,571	37,643
Term Loans Amortized Cost Basis by Origination Year, 2020	32,259	32,259	62,131
Term Loans Amortized Cost Basis by Origination Year, 2019	48,602	48,602	36,398
Term Loans Amortized Cost Basis by Origination Year, Pre-2019	60,740	60,740	33,953
Total	631,925	631,925	701,408
Commercial - Purchase | Performing
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2023			273,950
Term Loans Amortized Cost Basis by Origination Year, 2022	248,153	248,153	249,100
Term Loans Amortized Cost Basis by Origination Year, 2021	226,467	226,467	36,064
Term Loans Amortized Cost Basis by Origination Year, 2020	31,692	31,692	56,322
Term Loans Amortized Cost Basis by Origination Year, 2019	43,829	43,829	33,193
Term Loans Amortized Cost Basis by Origination Year, Pre-2019	53,563	53,563	30,208
Total	603,704	603,704	678,837
Commercial - Purchase | Nonperforming
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2023			1,274
Term Loans Amortized Cost Basis by Origination Year, 2022	9,600	9,600	6,959
Term Loans Amortized Cost Basis by Origination Year, 2021	6,104	6,104	1,579
Term Loans Amortized Cost Basis by Origination Year, 2020	567	567	5,809
Term Loans Amortized Cost Basis by Origination Year, 2019	4,773	4,773	3,205
Term Loans Amortized Cost Basis by Origination Year, Pre-2019	7,177	7,177	3,745
Total	28,221	28,221	22,571
Commercial - Refinance
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2023			272,766
Term Loans Amortized Cost Basis by Origination Year, 2022	253,514	253,514	222,699
Term Loans Amortized Cost Basis by Origination Year, 2021	202,891	202,891	59,650
Term Loans Amortized Cost Basis by Origination Year, 2020	52,090	52,090	127,056
Term Loans Amortized Cost Basis by Origination Year, 2019	106,986	106,986	113,569
Term Loans Amortized Cost Basis by Origination Year, Pre-2019	184,335	184,335	111,357
Total	799,816	799,816	907,097
Commercial - Refinance | Performing
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2023			263,754
Term Loans Amortized Cost Basis by Origination Year, 2022	233,052	233,052	210,898
Term Loans Amortized Cost Basis by Origination Year, 2021	188,723	188,723	55,795
Term Loans Amortized Cost Basis by Origination Year, 2020	47,883	47,883	103,633
Term Loans Amortized Cost Basis by Origination Year, 2019	92,819	92,819	93,161
Term Loans Amortized Cost Basis by Origination Year, Pre-2019	150,449	150,449	92,723
Total	712,926	712,926	819,964
Commercial - Refinance | Nonperforming
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2023			9,012
Term Loans Amortized Cost Basis by Origination Year, 2022	20,462	20,462	11,801
Term Loans Amortized Cost Basis by Origination Year, 2021	14,168	14,168	3,855
Term Loans Amortized Cost Basis by Origination Year, 2020	4,207	4,207	23,423
Term Loans Amortized Cost Basis by Origination Year, 2019	14,167	14,167	20,408
Term Loans Amortized Cost Basis by Origination Year, Pre-2019	33,886	33,886	18,634
Total	86,890	86,890	87,133
Residential 1-4 Unit - Purchase
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2023			256,906
Term Loans Amortized Cost Basis by Origination Year, 2022	225,743	225,743	237,342
Term Loans Amortized Cost Basis by Origination Year, 2021	208,850	208,850	12,875
Term Loans Amortized Cost Basis by Origination Year, 2020	10,282	10,282	33,998
Term Loans Amortized Cost Basis by Origination Year, 2019	25,850	25,850	20,444
Term Loans Amortized Cost Basis by Origination Year, Pre-2019	40,355	40,355	26,868
Total	511,080	511,080	588,433
Residential 1-4 Unit - Purchase | Performing
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2023			249,625
Term Loans Amortized Cost Basis by Origination Year, 2022	208,456	208,456	227,235
Term Loans Amortized Cost Basis by Origination Year, 2021	198,110	198,110	10,710
Term Loans Amortized Cost Basis by Origination Year, 2020	9,581	9,581	31,685
Term Loans Amortized Cost Basis by Origination Year, 2019	24,429	24,429	18,891
Term Loans Amortized Cost Basis by Origination Year, Pre-2019	34,251	34,251	22,303
Total	474,827	474,827	560,449
Residential 1-4 Unit - Purchase | Nonperforming
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2023			7,281
Term Loans Amortized Cost Basis by Origination Year, 2022	17,287	17,287	10,107
Term Loans Amortized Cost Basis by Origination Year, 2021	10,740	10,740	2,165
Term Loans Amortized Cost Basis by Origination Year, 2020	701	701	2,313
Term Loans Amortized Cost Basis by Origination Year, 2019	1,421	1,421	1,553
Term Loans Amortized Cost Basis by Origination Year, Pre-2019	6,104	6,104	4,565
Total	36,253	36,253	27,984
Residential 1-4 Unit - Refinance
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2023			360,350
Term Loans Amortized Cost Basis by Origination Year, 2022	321,252	321,252	310,218
Term Loans Amortized Cost Basis by Origination Year, 2021	273,503	273,503	31,610
Term Loans Amortized Cost Basis by Origination Year, 2020	27,587	27,587	111,954
Term Loans Amortized Cost Basis by Origination Year, 2019	89,388	89,388	55,704
Term Loans Amortized Cost Basis by Origination Year, Pre-2019	96,419	96,419	69,469
Total	808,149	808,149	939,305
Residential 1-4 Unit - Refinance | Performing
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2023			338,959
Term Loans Amortized Cost Basis by Origination Year, 2022	277,980	277,980	285,195
Term Loans Amortized Cost Basis by Origination Year, 2021	237,159	237,159	24,703
Term Loans Amortized Cost Basis by Origination Year, 2020	19,752	19,752	84,208
Term Loans Amortized Cost Basis by Origination Year, 2019	61,136	61,136	39,870
Term Loans Amortized Cost Basis by Origination Year, Pre-2019	74,197	74,197	52,461
Total	670,224	670,224	825,396
Residential 1-4 Unit - Refinance | Nonperforming
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2023			21,391
Term Loans Amortized Cost Basis by Origination Year, 2022	43,272	43,272	25,023
Term Loans Amortized Cost Basis by Origination Year, 2021	36,344	36,344	6,907
Term Loans Amortized Cost Basis by Origination Year, 2020	7,835	7,835	27,746
Term Loans Amortized Cost Basis by Origination Year, 2019	28,252	28,252	15,834
Term Loans Amortized Cost Basis by Origination Year, Pre-2019	22,222	22,222	17,008
Total	137,925	137,925	113,909
Short Term 1-4 Unit - Purchase
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2023			42,254
Term Loans Amortized Cost Basis by Origination Year, 2022	16,991	16,991	6,156
Term Loans Amortized Cost Basis by Origination Year, 2021	345	345	16,654
Term Loans Amortized Cost Basis by Origination Year, 2020	19,214	19,214	4,716
Term Loans Amortized Cost Basis by Origination Year, 2019	4,561	4,561	104
Total	41,111	41,111	69,884
Short Term 1-4 Unit - Purchase | Performing
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2023			40,967
Term Loans Amortized Cost Basis by Origination Year, 2022	11,458	11,458	944
Term Loans Amortized Cost Basis by Origination Year, 2021	180	180	15,659
Term Loans Amortized Cost Basis by Origination Year, 2020	18,510	18,510	4,174
Term Loans Amortized Cost Basis by Origination Year, 2019	4,561	4,561
Total	34,709	34,709	61,744
Short Term 1-4 Unit - Purchase | Nonperforming
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2023			1,287
Term Loans Amortized Cost Basis by Origination Year, 2022	5,533	5,533	5,212
Term Loans Amortized Cost Basis by Origination Year, 2021	165	165	995
Term Loans Amortized Cost Basis by Origination Year, 2020	704	704	542
Term Loans Amortized Cost Basis by Origination Year, 2019			104
Total	6,402	6,402	8,140
Short Term 1-4 Unit - Refinance
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2023			36,341
Term Loans Amortized Cost Basis by Origination Year, 2022	15,461	15,461	1,221
Term Loans Amortized Cost Basis by Origination Year, 2021	153	153	10,545
Term Loans Amortized Cost Basis by Origination Year, 2020	7,435	7,435	18,245
Term Loans Amortized Cost Basis by Origination Year, 2019	13,612	13,612	4,805
Term Loans Amortized Cost Basis by Origination Year, Pre-2019	4,150	4,150
Total	40,811	40,811	71,157
Short Term 1-4 Unit - Refinance | Performing
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2023			35,555
Term Loans Amortized Cost Basis by Origination Year, 2022	11,148	11,148
Total	11,148	11,148	35,555
Short Term 1-4 Unit - Refinance | Nonperforming
Financing Receivable Recorded Investment [Line Items]
Term Loans Amortized Cost Basis by Origination Year, 2023			786
Term Loans Amortized Cost Basis by Origination Year, 2022	4,313	4,313	1,221
Term Loans Amortized Cost Basis by Origination Year, 2021	153	153	10,545
Term Loans Amortized Cost Basis by Origination Year, 2020	7,435	7,435	18,245
Term Loans Amortized Cost Basis by Origination Year, 2019	13,612	13,612	4,805
Term Loans Amortized Cost Basis by Origination Year, Pre-2019	4,150	4,150
Total	$ 29,663	$ 29,663	$ 35,602