Loans - Loan Portfolio by Class According to Credit Quality Indicators (Details) - USD ($) $ in Thousands	Jun. 30, 2018	Dec. 31, 2017
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	$ 1,231,622	$ 1,147,864
SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	41,351	43,999
SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	41,351	43,999
SBA loans held for investment | Pass
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	39,215	42,415
SBA loans held for investment | Special mention
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	564	373
SBA loans held for investment | Substandard
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,572	1,211
SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	23,927	21,871
SBA 504 loans | Pass
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	22,838	20,751
SBA 504 loans | Special mention
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	0	1,024
SBA 504 loans | Substandard
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,089	96
Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	673,480	628,865
Commercial loans | Pass
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	666,704	622,014
Commercial loans | Special mention
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	2,302	4,670
Commercial loans | Substandard
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	4,474	2,181
Commercial loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	96,759	82,825
Commercial loans | Other | Pass
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	96,121	82,201
Commercial loans | Other | Special mention
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	187	599
Commercial loans | Other | Substandard
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	451	25
Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	479,530	469,696
Commercial loans | Commercial real estate | Pass
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	474,481	464,589
Commercial loans | Commercial real estate | Special mention
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	2,115	3,047
Commercial loans | Commercial real estate | Substandard
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	2,934	2,060
Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	73,264	54,473
Commercial loans | Commercial real estate construction | Pass
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	73,264	54,473
Commercial loans | Commercial real estate construction | Special mention
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	0	0
Commercial loans | Commercial real estate construction | Substandard
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	0	0
Total SBA, SBA 504 and commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	714,831	672,864
Total SBA, SBA 504 and commercial loans | Pass
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	705,919	664,429
Total SBA, SBA 504 and commercial loans | Special mention
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	2,866	5,043
Total SBA, SBA 504 and commercial loans | Substandard
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	6,046	3,392
Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	398,383	365,145
Residential mortgage loans | Performing
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	396,723	363,476
Residential mortgage loans | Nonperforming
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,660	1,669
Consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	118,408	109,855
Consumer loans | Performing
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	118,341	109,230
Consumer loans | Nonperforming
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	67	625
Consumer loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	57,434	54,038
Consumer loans | Other | Performing
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	57,434	54,038
Consumer loans | Other | Nonperforming
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	0	0
Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	60,974	55,817
Consumer loans | Home equity | Performing
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	60,907	55,192
Consumer loans | Home equity | Nonperforming
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	67	625
Total residential mortgage and consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	516,791	475,000
Total residential mortgage and consumer loans | Performing
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	515,064	472,706
Total residential mortgage and consumer loans | Nonperforming
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	$ 1,727	$ 2,294