Loans - Aging Analysis of Past Due and Nonaccrual Loans by Class (Details) - USD ($) $ in Thousands	Sep. 30, 2021	Dec. 31, 2020
Financing Receivable, Recorded Investment [Line Items]
Total past due	$ 26,474	$ 28,573
Financing receivable, nonaccrual status	8,802	12,060
Current	1,638,417	1,598,796
Total loans held for investment, current	1,618,287	1,589,909
Total	1,644,761	1,618,482
SBA loans held for sale	20,130	9,335
Total loans	1,664,891	1,627,817
Total loans	26,474	29,021
SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Current	22,371	19,681
Total	22,371	19,681
30-59 days past due
Financing Receivable, Recorded Investment [Line Items]
Total past due	12,573	8,882
Total loans	12,573	9,330
60-89 days past due
Financing Receivable, Recorded Investment [Line Items]
Total past due	2,835	7,182
90 days and still accruing
Financing Receivable, Recorded Investment [Line Items]
Total past due	2,264	449
SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total past due	1,957	4,545
Financing receivable, nonaccrual status	660	2,473
Current	36,029	35,042
Total	37,986	39,587
SBA loans held for investment | 30-59 days past due
Financing Receivable, Recorded Investment [Line Items]
Total past due	1,297	792
SBA loans held for investment | 60-89 days past due
Financing Receivable, Recorded Investment [Line Items]
Total past due		1,280
SBA PPP loans
Financing Receivable, Recorded Investment [Line Items]
Current	81,907	118,257
Total	81,907	118,257
Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total	924,037	839,788
Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total past due	5,040	6,139
Financing receivable, nonaccrual status	785	1,059
Current	705,829	624,284
Total	710,869	630,423
Commercial loans | Construction
Financing Receivable, Recorded Investment [Line Items]
Total past due		1,047
Current	74,336	70,357
Total	74,336	71,404
Commercial loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total past due	2,291	653
Financing receivable, nonaccrual status	2,094	266
Current	114,170	117,627
Total	116,461	118,280
Commercial loans | 30-59 days past due | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total past due	3,686	3,109
Commercial loans | 30-59 days past due | Construction
Financing Receivable, Recorded Investment [Line Items]
Total past due		1,047
Commercial loans | 30-59 days past due | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total past due	69	186
Commercial loans | 60-89 days past due | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total past due	569	1,971
Commercial loans | 60-89 days past due | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total past due		201
Commercial loans | 90 days and still accruing | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total past due	128
Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total past due	11,954	11,644
Financing receivable, nonaccrual status	2,626	5,217
Current	408,881	455,942
Total	420,835	467,586
Residential mortgage loans | 30-59 days past due
Financing Receivable, Recorded Investment [Line Items]
Total past due	5,102	3,232
Residential mortgage loans | 60-89 days past due
Financing Receivable, Recorded Investment [Line Items]
Total past due	2,266	2,933
Residential mortgage loans | 90 days and still accruing
Financing Receivable, Recorded Investment [Line Items]
Total past due	1,960	262
Consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total	71,071	66,100
Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total past due	34	1,875
Financing receivable, nonaccrual status		1,295
Current	68,575	60,674
Total	68,609	62,549
Consumer loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total past due		4
Current	2,462	3,547
Total	2,462	3,551
Consumer loans | 30-59 days past due | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total past due	34	393
Consumer loans | 30-59 days past due | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total past due		3
Consumer loans | 60-89 days past due | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total past due		1
Consumer loans | 90 days and still accruing | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total past due		187
Residential construction loans
Financing Receivable, Recorded Investment [Line Items]
Total past due	5,198	2,666
Financing receivable, nonaccrual status	2,637	1,750
Current	103,727	84,498
Total	108,925	87,164
Residential construction loans | 30-59 days past due
Financing Receivable, Recorded Investment [Line Items]
Total past due	2,385	120
Residential construction loans | 60-89 days past due
Financing Receivable, Recorded Investment [Line Items]
Total past due		796
Residential construction loans | 90 days and still accruing
Financing Receivable, Recorded Investment [Line Items]
Total past due	176
SBA loans held for sale
Financing Receivable, Recorded Investment [Line Items]
Total past due		448
Current	20,130	8,887
SBA loans held for sale	$ 20,130	9,335
SBA loans held for sale | 30-59 days past due
Financing Receivable, Recorded Investment [Line Items]
Total past due		$ 448