Loans - Loan Portfolio by Class According to Credit Quality Indicators (Details) - USD ($) $ in Thousands	Sep. 30, 2018	Dec. 31, 2017
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	$ 1,270,275	$ 1,147,864
SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	41,051	43,999
SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	41,051	43,999
SBA loans held for investment | Pass
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	39,092	42,415
SBA loans held for investment | Special mention
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	863	373
SBA loans held for investment | Substandard
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,096	1,211
SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	29,530	21,871
SBA 504 loans | Pass
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	28,458	20,751
SBA 504 loans | Special mention
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	0	1,024
SBA 504 loans | Substandard
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,072	96
Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	697,075	628,865
Commercial loans | Pass
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	690,020	622,014
Commercial loans | Special mention
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	2,263	4,670
Commercial loans | Substandard
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	4,792	2,181
Commercial loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	102,555	82,825
Commercial loans | Other | Pass
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	101,909	82,201
Commercial loans | Other | Special mention
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	187	599
Commercial loans | Other | Substandard
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	459	25
Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	516,634	469,696
Commercial loans | Commercial real estate | Pass
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	511,297	464,589
Commercial loans | Commercial real estate | Special mention
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	2,076	3,047
Commercial loans | Commercial real estate | Substandard
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	3,261	2,060
Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	48,356	54,473
Commercial loans | Commercial real estate construction | Pass
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	48,356	54,473
Commercial loans | Commercial real estate construction | Special mention
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	0	0
Commercial loans | Commercial real estate construction | Substandard
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	0	0
Total SBA, SBA 504 and commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	738,126	672,864
Total SBA, SBA 504 and commercial loans | Pass
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	729,112	664,429
Total SBA, SBA 504 and commercial loans | Special mention
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	3,126	5,043
Total SBA, SBA 504 and commercial loans | Substandard
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	5,888	3,392
Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	413,652	365,145
Residential mortgage loans | Performing
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	410,116	363,476
Residential mortgage loans | Nonperforming
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	3,536	1,669
Consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	118,497	109,855
Consumer loans | Performing
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	118,369	109,230
Consumer loans | Nonperforming
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	128	625
Consumer loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	58,794	54,038
Consumer loans | Other | Performing
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	58,794	54,038
Consumer loans | Other | Nonperforming
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	0	0
Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	59,703	55,817
Consumer loans | Home equity | Performing
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	59,575	55,192
Consumer loans | Home equity | Nonperforming
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	128	625
Total residential mortgage and consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	532,149	475,000
Total residential mortgage and consumer loans | Performing
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	528,485	472,706
Total residential mortgage and consumer loans | Nonperforming
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	$ 3,664	$ 2,294