Loans (Loan Portfolio by Class According to Credit Quality Indicators) (Details) - USD ($) $ in Thousands	Dec. 31, 2016	Dec. 31, 2015
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	$ 958,641	$ 875,844
SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	42,492	39,393
SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	26,344	29,353
Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	509,171	465,518
Commercial loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	58,447	49,332
Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	422,418	391,071
Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	28,306	25,115
SBA, SBA 504, and Commercial Loans [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	578,007	534,264
Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	289,093	264,523
Consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	91,541	77,057
Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	47,411	45,042
Consumer loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	44,130	32,015
Residential And Consumer Loans [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	380,634	341,580
Pass | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	38,990	35,032
Pass | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	24,635	24,003
Pass | Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	492,844	439,441
Pass | Commercial loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	57,000	45,870
Pass | Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	408,288	369,510
Pass | Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	27,556	24,061
Pass | SBA, SBA 504, and Commercial Loans [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	556,469	498,476
Special mention | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	2,023	2,647
Special mention | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,073	4,917
Special mention | Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	15,901	22,405
Special mention | Commercial loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,422	2,373
Special mention | Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	13,729	18,978
Special mention | Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	750	1,054
Special mention | SBA, SBA 504, and Commercial Loans [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	18,997	29,969
Substandard | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,479	1,714
Substandard | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	636	433
Substandard | Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	426	3,672
Substandard | Commercial loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	25	1,089
Substandard | Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	401	2,583
Substandard | Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	0	0
Substandard | SBA, SBA 504, and Commercial Loans [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	2,541	5,819
Performing | Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	286,421	262,299
Performing | Consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	890,833	76,467
Performing | Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	46,929	44,452
Performing | Consumer loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	42,154	32,015
Performing | Residential And Consumer Loans [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	375,504	338,766
Nonperforming | Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	2,672	2,224
Nonperforming | Consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	2,458	590
Nonperforming | Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	482	590
Nonperforming | Consumer loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,976	0
Nonperforming | Residential And Consumer Loans [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	$ 5,130	$ 2,814