Loans - Narrative (Details)	12 Months Ended
	Dec. 31, 2022 USD ($) PSARate payment Loan qtr	Dec. 31, 2021 USD ($) PSARate Loan	Dec. 31, 2020 USD ($)
Accounts, Notes, Loans and Financing Receivable [Line Items]
Deferred loan costs	$ 26,600,000	$ 22,400,000
Proceeds from sale of loans held-for-investment	63,397,000	10,032,000	$ 2,395,000
Gain (loss) on sale of loans	6,431,000	35,880,000	62,560,000
Accrued Interest	200,000	200,000	500,000
Interest income	0	0	0
Troubled debt restructuring, specific reserve	3,700,000	3,600,000
Additional amounts committed to lend as troubled debt restructurings	40,000.00	40,000.00
Charge offs due to impaired trouble debt restructurings	$ 0		0
Past due period for modified loans	90 days
Number of consecutive payments for mortgage loans missed | payment	4
Charge-offs on TDRs that subsequently defaulted	$ 0	0	0
Interest and fees on loans	139,057,000	116,644,000	123,159,000
Custodial deposit accounts	$ 28,400,000	$ 39,400,000
Maximum number of consecutive quarterly losses recorded for profitability requirement | qtr	4
Percentage of decline in net worth during the period	30.00%
Maximum percentage of decline in net worth for one consecutive quarter	25.00%
Maximum percentage of decline in net worth for two consecutive quarters	40.00%
Average coupon rate	3.60%	3.46%
Average servicing fee	0.26%	0.256%
Average discount rate	10.12%	10.07%
Average PSA rate | PSARate	133	232
Loans outstanding, net	$ 3,412,917,000	$ 2,857,793,000
Allowance for credit loss, period increase (decrease)	30,000.00		40,000.00
Paycheck Protection Program
Accounts, Notes, Loans and Financing Receivable [Line Items]
Allowance for Credit Losses	$ 0
Number of outstanding loans | Loan	0	186,000
Unaccreted net fees	$ 0	$ 800,000
Loans funded	0	135,500,000	261,100,000
Interest and fees on loans	$ 800,000	8,900,000	5,600,000
Total loans, principal balance		26,400,000
Minimum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Modification of stated interest rate of loans, range of period	9 months
Modifications involving extension of maturity date, period range	1 month
Maximum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Modification of stated interest rate of loans, range of period	36 months
Modification of stated interest rate of loans, range of period in certain circumstances	480 months
Modifications involving extension of maturity date, period range	60 months
Modifications involving extension of maturity date, period range in certain circumstances	230 months
Fixed Rate Residential Mortgage
Accounts, Notes, Loans and Financing Receivable [Line Items]
Proceeds from sale of loans held-for-investment	$ 63,400,000	9,600,000
Gain (loss) on sale of loans	$ 550,000	$ 450,000
Mortgage | Fixed Rate Residential Mortgage | Freddie Mac
Accounts, Notes, Loans and Financing Receivable [Line Items]
Proceeds from sale of loans held-for-investment			26,300,000
Gain (loss) on sale of loans			720,000
Mortgage | Residential Adjustable Rate Mortgage
Accounts, Notes, Loans and Financing Receivable [Line Items]
Proceeds from sale of loans held-for-investment			2,400,000
Gain (loss) on sale of loans			$ 70,000.00