Loans - Aging Analysis of Past Due and Nonaccrual Loans by Class (Details) - USD ($) $ in Thousands	Sep. 30, 2023	Dec. 31, 2022
Financing Receivable, Recorded Investment [Line Items]
Total loans		$ 2,106,559
Total loans	$ 2,173,190
SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans		35,077
Total loans	31,808
SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	40,873	38,468
Total loans		38,468
Total loans	40,873
SBA loans held for investment | 30-59 days past due
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	4,083
SBA loans held for investment | 60-89 days past due
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	420	576
SBA loans held for investment | Nonaccrual
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	3,445	690
SBA loans held for investment | Past Due
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	7,948	1,266
SBA loans held for investment | Current
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	32,925	37,202
SBA PPP loans
Financing Receivable, Recorded Investment [Line Items]
Total loans		5,908
Total loans	2,507
Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans		1,187,543
Total loans	1,276,156
Commercial loans | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	31,808	35,077
Total loans		35,077
Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	968,801	903,126
Total loans		903,126
Total loans	968,801
Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	144,580	131,774
Total loans		131,774
Total loans	144,580
Commercial loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	130,967	117,566
Total loans		117,566
Total loans	130,967
Commercial loans | 30-59 days past due | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	553	22
Commercial loans | 30-59 days past due | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP		198
Commercial loans | 60-89 days past due | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP		188
Commercial loans | 60-89 days past due | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP		300
Commercial loans | Nonaccrual | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	82	805
Commercial loans | Nonaccrual | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	668	777
Commercial loans | Past Due | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	635	1,015
Commercial loans | Past Due | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	668	1,275
Commercial loans | Current | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	31,808	35,077
Commercial loans | Current | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	968,166	902,111
Commercial loans | Current | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	144,580	131,774
Commercial loans | Current | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	130,299	116,291
Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	628,628	605,091
Total loans		605,091
Total loans	628,628
Residential mortgage loans | 30-59 days past due
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	4,961
Residential mortgage loans | 60-89 days past due
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	3,347	982
Residential mortgage loans | 90 days and still accruing
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	260
Residential mortgage loans | Nonaccrual
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	10,270	3,361
Residential mortgage loans | Past Due
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	18,838	4,343
Residential mortgage loans | Current
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	609,790	600,748
Consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans		78,164
Total loans	72,189
Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	65,461	68,310
Total loans		68,310
Total loans	65,461
Consumer loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	6,728	9,854
Total loans		9,854
Total loans	6,728
Consumer loans | 30-59 days past due | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	286
Consumer loans | 30-59 days past due | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	1,319	18
Consumer loans | 60-89 days past due | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	15
Consumer loans | 60-89 days past due | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	52	7
Consumer loans | 90 days and still accruing | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	5
Consumer loans | Nonaccrual | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	125
Consumer loans | Past Due | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	426
Consumer loans | Past Due | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	1,376	25
Consumer loans | Current | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	65,035	68,310
Consumer loans | Current | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	5,352	9,829
Residential construction loans
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	133,450	163,457
Total loans		163,457
Total loans	133,450
Residential construction loans | Nonaccrual
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	3,201	3,432
Residential construction loans | Past Due
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	3,201	3,432
Residential construction loans | Current
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	130,249	160,025
Residential mortgage, consumer and residential construction loans
Financing Receivable, Recorded Investment [Line Items]
Total loans		846,712
Total loans held for investment including PPP
Financing Receivable, Recorded Investment [Line Items]
Total loans		2,078,631
Total loans	2,153,803
Total loans held for investment excluding SBA PPP
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	2,151,296	2,072,723
Total loans	2,151,296
Total loans held for investment excluding SBA PPP | 30-59 days past due
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	11,202	238
Total loans held for investment excluding SBA PPP | 60-89 days past due
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	3,834	2,053
Total loans held for investment excluding SBA PPP | 90 days and still accruing
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	265
Total loans held for investment excluding SBA PPP | Nonaccrual
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	17,791	9,065
Total loans held for investment excluding SBA PPP | Past Due
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	33,092	11,356
Total loans held for investment excluding SBA PPP | Current
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	2,118,204	2,061,367
SBA loans held for sale
Financing Receivable, Recorded Investment [Line Items]
Total loans		27,928
Total loans	19,387
SBA loans held for sale | 30-59 days past due
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	815	2,195
SBA loans held for sale | Past Due
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	815	2,195
SBA loans held for sale | Current
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	18,572	25,733
Total loans excluding SBA PPP loans
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	2,170,683	2,100,651
Total loans excluding SBA PPP loans | 30-59 days past due
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	12,017	2,433
Total loans excluding SBA PPP loans | 60-89 days past due
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	3,834	2,053
Total loans excluding SBA PPP loans | 90 days and still accruing
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	265
Total loans excluding SBA PPP loans | Nonaccrual
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	17,791	9,065
Total loans excluding SBA PPP loans | Past Due
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	33,907	13,551
Total loans excluding SBA PPP loans | Current
Financing Receivable, Recorded Investment [Line Items]
Total loans, excluding SBA PPP	$ 2,136,776	$ 2,087,100