Loans (Loan Portfolio by Class According to Credit Quality Indicators) (Details) - USD ($) $ in Thousands	Dec. 31, 2018	Dec. 31, 2017
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	$ 1,293,395	$ 1,147,864
SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	39,333	43,999
SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	29,155	21,871
Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	694,102	628,865
Commercial loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	104,587	82,825
Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	510,370	469,696
Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	49,990	54,473
SBA, SBA 504, and Commercial Loans [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	733,435	672,864
Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	436,056	365,145
Consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	123,904	109,855
Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	59,887	55,817
Consumer loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	64,017	54,038
Residential And Consumer Loans [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	559,960	475,000
Pass | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	37,198	42,415
Pass | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	28,105	20,751
Pass | Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	685,923	622,014
Pass | Commercial loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	103,806	82,201
Pass | Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	504,022	464,589
Pass | Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	49,990	54,473
Pass | SBA, SBA 504, and Commercial Loans [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	723,121	664,429
Special mention | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	601	373
Special mention | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	0	1,024
Special mention | Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	3,201	4,670
Special mention | Commercial loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	322	599
Special mention | Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	2,879	3,047
Special mention | Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	0	0
Special mention | SBA, SBA 504, and Commercial Loans [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	3,802	5,043
Substandard | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,534	1,211
Substandard | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,050	96
Substandard | Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	4,978	2,181
Substandard | Commercial loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	459	25
Substandard | Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	3,469	2,060
Substandard | Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	0	0
Substandard | SBA, SBA 504, and Commercial Loans [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	6,512	3,392
Performing | Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	431,845	363,476
Performing | Consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	123,878	109,230
Performing | Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	59,861	55,192
Performing | Consumer loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	64,017	54,038
Performing | Residential And Consumer Loans [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	555,723	472,706
Nonperforming | Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	4,211	1,669
Nonperforming | Consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	26	625
Nonperforming | Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	26	625
Nonperforming | Consumer loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	0	0
Nonperforming | Residential And Consumer Loans [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	$ 4,237	$ 2,294