Loans - Credit Quality Indicators (Details) - USD ($) $ in Thousands	Mar. 31, 2020	Dec. 31, 2019
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	$ 1,428,919	$ 1,412,029
SBA loans held for investment
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	37,074	35,767
SBA loans held for investment | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	35,994	34,202
SBA loans held for investment | Special mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	0	1,115
SBA loans held for investment | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	1,080	450
SBA 504 loans
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	25,185	26,726
SBA 504 loans | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	23,374	24,878
SBA 504 loans | Special mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	1,779	1,808
SBA 504 loans | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	32	40
Commercial loans
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	786,078	765,032
Commercial loans | Other
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	120,029	112,014
Commercial loans | Commercial real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	586,336	578,643
Commercial loans | Commercial real estate construction
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	54,528	47,649
Commercial loans | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	776,786	756,073
Commercial loans | Pass | Other
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	114,840	107,220
Commercial loans | Pass | Commercial real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	584,044	576,326
Commercial loans | Pass | Commercial real estate construction
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	54,528	47,649
Commercial loans | Special mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	6,053	5,927
Commercial loans | Special mention | Other
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	3,525	3,361
Commercial loans | Special mention | Commercial real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	749	758
Commercial loans | Special mention | Commercial real estate construction
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	0	0
Commercial loans | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	3,239	3,032
Commercial loans | Substandard | Other
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	1,664	1,433
Commercial loans | Substandard | Commercial real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	1,543	1,559
Commercial loans | Substandard | Commercial real estate construction
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	0	0
Total SBA, SBA 504 and commercial loans
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	823,152	800,799
Total SBA, SBA 504 and commercial loans | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	812,780	790,275
Total SBA, SBA 504 and commercial loans | Special mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	6,053	7,042
Total SBA, SBA 504 and commercial loans | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	4,319	3,482
Residential mortgage loans
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	456,072	467,706
Residential mortgage loans | Performing
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	449,162	463,770
Residential mortgage loans | Nonperforming
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	6,910	3,936
Consumer loans
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	149,695	143,524
Consumer loans | Other
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	74,455	73,935
Consumer loans | Home equity
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	75,240	69,589
Consumer loans | Performing
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	149,190	143,504
Consumer loans | Performing | Other
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	74,455	73,915
Consumer loans | Performing | Home equity
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	74,735	69,589
Consumer loans | Nonperforming
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	505	20
Consumer loans | Nonperforming | Other
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	0	20
Consumer loans | Nonperforming | Home equity
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	505	0
Total residential mortgage and consumer loans
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	605,767	611,230
Total residential mortgage and consumer loans | Performing
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	598,352	607,274
Total residential mortgage and consumer loans | Nonperforming
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	$ 7,415	$ 3,956