Loans - Credit Quality Indicators (Details) - USD ($) $ in Thousands	Mar. 31, 2019	Dec. 31, 2018
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	$ 1,304,984	$ 1,293,395
SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	38,815	39,333
SBA loans held for investment | Pass
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	36,750	37,198
SBA loans held for investment | Special mention
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,315	601
SBA loans held for investment | Substandard
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	750	1,534
SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	27,129	29,155
SBA 504 loans | Pass
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	26,093	28,105
SBA 504 loans | Special mention
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	0	0
SBA 504 loans | Substandard
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,036	1,050
Commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	702,235	694,102
Commercial loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	98,775	104,587
Commercial loans | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	522,511	510,370
Commercial loans | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	53,820	49,990
Commercial loans | Pass
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	692,394	685,923
Commercial loans | Pass | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	96,261	103,806
Commercial loans | Pass | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	516,220	504,022
Commercial loans | Pass | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	53,820	49,990
Commercial loans | Special mention
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	6,808	3,201
Commercial loans | Special mention | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	2,514	322
Commercial loans | Special mention | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	4,294	2,879
Commercial loans | Special mention | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	0	0
Commercial loans | Substandard
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	3,033	4,978
Commercial loans | Substandard | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	0	459
Commercial loans | Substandard | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,997	3,469
Commercial loans | Substandard | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	0	0
Total SBA, SBA 504 and commercial loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	741,050	733,435
Total SBA, SBA 504 and commercial loans | Pass
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	729,144	723,121
Total SBA, SBA 504 and commercial loans | Special mention
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	8,123	3,802
Total SBA, SBA 504 and commercial loans | Substandard
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	3,783	6,512
Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	438,431	436,056
Residential mortgage loans | Performing
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	433,188	431,845
Residential mortgage loans | Nonperforming
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	5,243	4,211
Consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	125,503	123,904
Consumer loans | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	64,558	64,017
Consumer loans | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	60,945	59,887
Consumer loans | Performing
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	125,332	123,878
Consumer loans | Performing | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	64,558	64,017
Consumer loans | Performing | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	60,774	59,861
Consumer loans | Nonperforming
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	171	26
Consumer loans | Nonperforming | Other
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	0	0
Consumer loans | Nonperforming | Home equity
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	171	26
Total residential mortgage and consumer loans
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	563,934	559,960
Total residential mortgage and consumer loans | Performing
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	558,520	555,723
Total residential mortgage and consumer loans | Nonperforming
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	$ 5,414	$ 4,237