Loans - Aging Analysis of Past Due and Nonaccrual Loans by Class (Details) - USD ($) $ in Thousands	Mar. 31, 2021	Dec. 31, 2020
Financing Receivable, Recorded Investment [Line Items]
Total past due	$ 29,726	$ 28,573
Financing receivable, nonaccrual status	11,788	12,060
Current	1,636,058	1,598,796
Total loans held for investment, current	1,629,913	1,589,909
Total	1,659,639	1,618,482
SBA loans held for sale	8,809	9,335
Total loans	1,668,448	1,627,817
Total loans	32,390	29,021
SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Current	21,255	19,681
Total	21,255	19,681
30-59 days past due
Financing Receivable, Recorded Investment [Line Items]
Total past due	12,911	8,882
Total loans	15,575	9,330
60-89 days past due
Financing Receivable, Recorded Investment [Line Items]
Total past due	2,487	7,182
90 days and still accruing
Financing Receivable, Recorded Investment [Line Items]
Total past due	2,540	449
SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total past due	7,901	4,545
Financing receivable, nonaccrual status	1,560	2,473
Current	30,395	35,042
Total	38,296	39,587
SBA loans held for investment | 30-59 days past due
Financing Receivable, Recorded Investment [Line Items]
Total past due	6,341	792
SBA loans held for investment | 60-89 days past due
Financing Receivable, Recorded Investment [Line Items]
Total past due		1,280
SBA PPP loans
Financing Receivable, Recorded Investment [Line Items]
Current	169,117	118,257
Total	169,117	118,257
Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total	853,078	839,788
Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total past due	2,288	6,139
Financing receivable, nonaccrual status	612	1,059
Current	648,581	624,284
Total	650,869	630,423
Commercial loans | Construction
Financing Receivable, Recorded Investment [Line Items]
Total past due		1,047
Current	66,972	70,357
Total	66,972	71,404
Commercial loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total past due	443	653
Financing receivable, nonaccrual status	340	266
Current	113,539	117,627
Total	113,982	118,280
Commercial loans | 30-59 days past due | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total past due	889	3,109
Commercial loans | 30-59 days past due | Construction
Financing Receivable, Recorded Investment [Line Items]
Total past due		1,047
Commercial loans | 30-59 days past due | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total past due	43	186
Commercial loans | 60-89 days past due | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total past due	787	1,971
Commercial loans | 60-89 days past due | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total past due	60	201
Consumer construction loans
Financing Receivable, Recorded Investment [Line Items]
Total past due	4,047	2,666
Financing receivable, nonaccrual status	2,565	1,750
Current	86,450	84,498
Total	90,497	87,164
Consumer construction loans | 30-59 days past due
Financing Receivable, Recorded Investment [Line Items]
Total past due	854	120
Consumer construction loans | 60-89 days past due
Financing Receivable, Recorded Investment [Line Items]
Total past due	416	796
Consumer construction loans | 90 days and still accruing
Financing Receivable, Recorded Investment [Line Items]
Total past due	212
Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total past due	14,808	11,644
Financing receivable, nonaccrual status	6,711	5,217
Current	433,341	455,942
Total	448,149	467,586
Residential mortgage loans | 30-59 days past due
Financing Receivable, Recorded Investment [Line Items]
Total past due	4,728	3,232
Residential mortgage loans | 60-89 days past due
Financing Receivable, Recorded Investment [Line Items]
Total past due	1,224	2,933
Residential mortgage loans | 90 days and still accruing
Financing Receivable, Recorded Investment [Line Items]
Total past due	2,145	262
Consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total	60,502	66,100
Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total past due	232	1,875
Financing receivable, nonaccrual status		1,295
Current	59,006	60,674
Total	59,238	62,549
Consumer loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total past due	7	4
Current	1,257	3,547
Total	1,264	3,551
Consumer loans | 30-59 days past due | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total past due	49	393
Consumer loans | 30-59 days past due | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total past due	7	3
Consumer loans | 60-89 days past due | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total past due		1
Consumer loans | 90 days and still accruing | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total past due	183	187
Total SBA, SBA 504 and commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total	1,060,491	997,632
SBA loans held for sale
Financing Receivable, Recorded Investment [Line Items]
Total past due	2,664	448
Current	6,145	8,887
SBA loans held for sale	8,809	9,335
SBA loans held for sale | 30-59 days past due
Financing Receivable, Recorded Investment [Line Items]
Total past due	2,664	448
Pass | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total	21,255	19,681
Pass | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total	35,932	36,991
Pass | SBA PPP loans
Financing Receivable, Recorded Investment [Line Items]
Total	169,117	118,257
Pass | Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total	817,006	804,239
Pass | Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total	623,270	603,482
Pass | Commercial loans | Construction
Financing Receivable, Recorded Investment [Line Items]
Total	66,972	71,404
Pass | Commercial loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total	105,509	109,672
Pass | Total SBA, SBA 504 and commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total	1,022,055	959,487
Special mention | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total	532	525
Special mention | Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total	27,932	30,739
Special mention | Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total	22,519	25,206
Special mention | Commercial loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total	5,413	5,533
Special mention | Total SBA, SBA 504 and commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total	28,464	31,264
Substandard | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total	1,832	2,071
Substandard | Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total	8,140	4,810
Substandard | Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total	5,080	1,735
Substandard | Commercial loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total	3,060	3,075
Substandard | Total SBA, SBA 504 and commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total	9,972	6,881
Performing | Consumer construction loans
Financing Receivable, Recorded Investment [Line Items]
Total	87,932	85,414
Performing | Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total	441,438	462,369
Performing | Consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total	60,502	64,805
Performing | Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total	59,238	61,254
Performing | Consumer loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total	1,264	3,551
Nonperforming | Consumer construction loans
Financing Receivable, Recorded Investment [Line Items]
Total	2,565	1,750
Nonperforming | Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total	$ 6,711	5,217
Nonperforming | Consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total		1,295
Nonperforming | Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total		$ 1,295