Loans - Loan Portfolio by Class According to Credit Quality Indicators (Details) - USD ($) $ in Thousands	Sep. 30, 2016	Dec. 31, 2015
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	$ 934,221	$ 875,844
SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	41,795	39,393
SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	26,067	29,353
Commercial other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	52,927	49,332
Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	407,686	391,071
Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	35,395	25,115
Total commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	496,008	465,518
Total SBA, SBA 504 and commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	563,870	534,264
Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	282,317	264,523
Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	45,495	45,042
Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	42,539	32,015
Total consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	88,034	77,057
Total residential mortgage and consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	370,351	341,580
Pass | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	38,560	35,032
Pass | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	22,625	24,003
Pass | Commercial other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	50,724	45,870
Pass | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	394,051	369,510
Pass | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	34,339	24,061
Pass | Total commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	479,114	439,441
Pass | Total SBA, SBA 504 and commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	540,299	498,476
Special mention | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,640	2,647
Special mention | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	2,797	4,917
Special mention | Commercial other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,617	2,373
Special mention | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	13,609	18,978
Special mention | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	750	1,054
Special mention | Total commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	15,976	22,405
Special mention | Total SBA, SBA 504 and commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	20,413	29,969
Substandard | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,595	1,714
Substandard | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	645	433
Substandard | Commercial other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	586	1,089
Substandard | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	451	2,583
Substandard | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	306	0
Substandard | Total commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,343	3,672
Substandard | Total SBA, SBA 504 and commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	3,583	5,819
Performing | Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	279,719	262,299
Performing | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	45,006	44,452
Performing | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	42,539	32,015
Performing | Total consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	87,545	76,467
Performing | Total residential mortgage and consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	367,264	338,766
Nonperforming | Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	2,598	2,224
Nonperforming | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	489	590
Nonperforming | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	0	0
Nonperforming | Total consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	489	590
Nonperforming | Total residential mortgage and consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	$ 3,087	$ 2,814