Loans (Aging Analysis of Past Due and Nonaccrual Loans by Class) (Details) - USD ($) $ in Thousands	Dec. 31, 2019	Dec. 31, 2018
Financing Receivable, Recorded Investment [Line Items]
Total past due	$ 17,517	$ 12,973
Nonaccrual	5,649	6,873
Current	1,408,041	1,291,593
Total loans held for investment, current	1,394,512	1,280,422
Total	1,412,029	1,293,395
SBA loans held for sale	13,529	11,171
Total loans	1,425,558	1,304,566
SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total past due	2,212	1,560
Nonaccrual	1,164	1,560
Current	33,555	37,773
Total	35,767	39,333
SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total past due	1,808	0
Nonaccrual	0	0
Current	24,918	29,155
Total	26,726	29,155
Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total	765,032	694,102
Commercial loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total past due	387	30
Nonaccrual	316	30
Current	111,627	104,557
Total	112,014	104,587
Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total past due	428	1,347
Nonaccrual	213	1,046
Current	578,215	509,023
Total	578,643	510,370
Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	0
Nonaccrual	0	0
Current	47,649	49,990
Total	47,649	49,990
Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total past due	10,925	9,314
Nonaccrual	3,936	4,211
Current	456,781	426,742
Total	467,706	436,056
Consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total	143,524	123,904
Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total past due	1,624	422
Nonaccrual	0	26
Current	67,965	59,465
Total	69,589	59,887
Consumer loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total past due	133	300
Nonaccrual	20	0
Current	73,802	63,717
Total	73,935	64,017
30-59 days past due
Financing Receivable, Recorded Investment [Line Items]
Total past due	7,163	4,798
30-59 days past due | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total past due	1,048	0
30-59 days past due | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	0
30-59 days past due | Commercial loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total past due	71	0
30-59 days past due | Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total past due	215	301
30-59 days past due | Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	0
30-59 days past due | Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total past due	4,383	3,801
30-59 days past due | Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total past due	1,446	396
30-59 days past due | Consumer loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	300
60-89 days past due
Financing Receivable, Recorded Investment [Line Items]
Total past due	3,775	1,204
60-89 days past due | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	0
60-89 days past due | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total past due	1,808	0
60-89 days past due | Commercial loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	0
60-89 days past due | Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	0
60-89 days past due | Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	0
60-89 days past due | Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total past due	1,676	1,204
60-89 days past due | Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total past due	178	0
60-89 days past due | Consumer loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total past due	113	0
90 days and still accruing
Financing Receivable, Recorded Investment [Line Items]
Total past due	930	98
90 days and still accruing | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	0
90 days and still accruing | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	0
90 days and still accruing | Commercial loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	0
90 days and still accruing | Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	0
90 days and still accruing | Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	0
90 days and still accruing | Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total past due	930	98
90 days and still accruing | Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	0
90 days and still accruing | Consumer loans | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	0
Collateral Pledged | Small Business Administration
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual	$ 59	$ 89