Loans - Loan Portfolio by Class According to Credit Quality Indicators (Details) - USD ($) $ in Thousands	Jun. 30, 2016	Dec. 31, 2015
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	$ 901,798	$ 875,844
SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	40,006	39,393
SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	27,038	29,353
Commercial other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	51,981	49,332
Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	396,480	391,071
Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	33,252	25,115
Total commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	481,713	465,518
Total SBA, SBA 504 and commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	548,757	534,264
Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	268,774	264,523
Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	44,518	45,042
Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	39,749	32,015
Total consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	84,267	77,057
Total residential mortgage and consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	353,041	341,580
Pass | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	36,748	35,032
Pass | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	23,555	24,003
Pass | Commercial other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	48,721	45,870
Pass | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	380,332	369,510
Pass | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	32,225	24,061
Pass | Total commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	461,278	439,441
Pass | Total SBA, SBA 504 and commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	521,581	498,476
Special mention | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,435	2,647
Special mention | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	2,829	4,917
Special mention | Commercial other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,731	2,373
Special mention | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	15,659	18,978
Special mention | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	721	1,054
Special mention | Total commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	18,111	22,405
Special mention | Total SBA, SBA 504 and commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	22,375	29,969
Substandard | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,823	1,714
Substandard | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	654	433
Substandard | Commercial other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,529	1,089
Substandard | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	489	2,583
Substandard | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	306	0
Substandard | Total commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	2,324	3,672
Substandard | Total SBA, SBA 504 and commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	4,801	5,819
Performing | Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	267,185	262,299
Performing | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	44,019	44,452
Performing | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	39,749	32,015
Performing | Total consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	83,768	76,467
Performing | Total residential mortgage and consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	350,953	338,766
Nonperforming | Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,589	2,224
Nonperforming | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	499	590
Nonperforming | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	0	0
Nonperforming | Total consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	499	590
Nonperforming | Total residential mortgage and consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	$ 2,088	$ 2,814