Loans (Aging Analysis of Past Due and Nonaccrual Loans by Class) (Details) - USD ($) $ in Thousands	Dec. 31, 2018	Dec. 31, 2017
Financing Receivable, Recorded Investment [Line Items]
Total past due	$ 12,973	$ 8,308
Nonaccrual	6,873	2,994
Current	1,291,593	1,162,366
Total loans held for investment, current	1,280,422	1,139,556
Total	1,293,395	1,147,864
SBA loans held for sale	11,171	22,810
Total loans	1,304,566	1,170,674
SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total past due	1,560	1,185
Nonaccrual	1,560	632
Current	37,773	42,814
Total	39,333	43,999
SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	0
Nonaccrual	0	0
Current	29,155	21,871
Total	29,155	21,871
Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total	694,102	628,865
Commercial loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total past due	30	108
Nonaccrual	30	25
Current	104,557	82,717
Total	104,587	82,825
Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total past due	1,347	1,674
Nonaccrual	1,046	43
Current	509,023	468,022
Total	510,370	469,696
Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	0
Nonaccrual	0	0
Current	49,990	54,473
Total	49,990	54,473
Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total past due	9,314	4,457
Nonaccrual	4,211	1,669
Current	426,742	360,688
Total	436,056	365,145
Consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total	123,904	109,855
Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total past due	422	881
Nonaccrual	26	625
Current	59,465	54,936
Total	59,887	55,817
Consumer loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total past due	300	3
Nonaccrual	0	0
Current	63,717	54,035
Total	64,017	54,038
30-59 days past due
Financing Receivable, Recorded Investment [Line Items]
Total past due	4,798	2,705
30-59 days past due | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	240
30-59 days past due | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	0
30-59 days past due | Commercial loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	23
30-59 days past due | Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total past due	301	558
30-59 days past due | Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	0
30-59 days past due | Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total past due	3,801	1,830
30-59 days past due | Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total past due	396	51
30-59 days past due | Consumer loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total past due	300	3
60-89 days past due
Financing Receivable, Recorded Investment [Line Items]
Total past due	1,204	2,549
60-89 days past due | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	313
60-89 days past due | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	0
60-89 days past due | Commercial loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	0
60-89 days past due | Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	1,073
60-89 days past due | Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	0
60-89 days past due | Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total past due	1,204	958
60-89 days past due | Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	205
60-89 days past due | Consumer loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	0
90 days and still accruing
Financing Receivable, Recorded Investment [Line Items]
Total past due	98	60
90 days and still accruing | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	0
90 days and still accruing | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	0
90 days and still accruing | Commercial loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	60
90 days and still accruing | Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	0
90 days and still accruing | Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	0
90 days and still accruing | Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total past due	98	0
90 days and still accruing | Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	0
90 days and still accruing | Consumer loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total past due	0	0
Collateral Pledged | Small Business Administration
Financing Receivable, Recorded Investment [Line Items]
Nonaccrual	$ 89	$ 27