Loans - Credit Quality Indicators (Details) - USD ($) $ in Thousands	Mar. 31, 2017	Dec. 31, 2016
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	$ 988,514	$ 958,641
SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	42,403	42,492
SBA loans held for investment | Pass
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	39,549	38,990
SBA loans held for investment | Special mention
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,458	2,023
SBA loans held for investment | Substandard
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,396	1,479
SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	25,111	26,344
SBA 504 loans | Pass
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	23,698	24,635
SBA 504 loans | Special mention
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,061	1,073
SBA 504 loans | Substandard
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	352	636
Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	519,338	509,171
Commercial loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	60,381	58,447
Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	425,581	422,418
Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	33,376	28,306
Commercial loans | Pass
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	509,657	492,844
Commercial loans | Pass | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	59,115	57,000
Commercial loans | Pass | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	417,916	408,288
Commercial loans | Pass | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	32,626	27,556
Commercial loans | Special mention
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	7,806	15,901
Commercial loans | Special mention | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	293	1,422
Commercial loans | Special mention | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	6,763	13,729
Commercial loans | Special mention | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	750	750
Commercial loans | Substandard
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,875	426
Commercial loans | Substandard | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	973	25
Commercial loans | Substandard | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	902	401
Commercial loans | Substandard | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	0	0
Total SBA, SBA 504 and commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	586,852	578,007
Total SBA, SBA 504 and commercial loans | Pass
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	572,904	556,469
Total SBA, SBA 504 and commercial loans | Special mention
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	10,325	18,997
Total SBA, SBA 504 and commercial loans | Substandard
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	3,623	2,541
Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	305,578	289,093
Residential mortgage loans | Performing
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	303,064	286,421
Residential mortgage loans | Nonperforming
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	2,514	2,672
Consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	96,084	91,541
Consumer loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	46,789	44,130
Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	49,295	47,411
Consumer loans | Performing
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	93,815	89,083
Consumer loans | Performing | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	44,814	42,154
Consumer loans | Performing | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	49,001	46,929
Consumer loans | Nonperforming
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	2,269	2,458
Consumer loans | Nonperforming | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,975	1,976
Consumer loans | Nonperforming | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	294	482
Total residential mortgage and consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	401,662	380,634
Total residential mortgage and consumer loans | Performing
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	396,879	375,504
Total residential mortgage and consumer loans | Nonperforming
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	$ 4,783	$ 5,130