Loans - Credit Quality Indicators (Details) - USD ($) $ in Thousands	Jun. 30, 2020	Dec. 31, 2019
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	$ 1,581,905	$ 1,412,029
SBA 504 loans
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	23,299	26,726
Pass | SBA 504 loans
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	23,275	24,878
Special mention | SBA 504 loans
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment		1,808
Substandard | SBA 504 loans
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	24	40
SBA loans held for investment
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	36,966	35,767
SBA loans held for investment | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	35,691	34,202
SBA loans held for investment | Special mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment		1,115
SBA loans held for investment | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	1,275	450
SBA PPP loans
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	136,039	0
SBA PPP loans | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	136,007
SBA PPP loans | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	32
Commercial loans
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	792,752	765,032
Commercial loans | Other
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	121,961	112,014
Commercial loans | Commercial real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	586,332	578,643
Commercial loans | Commercial real estate construction
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	61,160	47,649
Commercial loans | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	771,047	756,073
Commercial loans | Pass | Other
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	117,793	107,220
Commercial loans | Pass | Commercial real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	568,819	576,326
Commercial loans | Pass | Commercial real estate construction
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	61,160	47,649
Commercial loans | Special mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	2,546	5,927
Commercial loans | Special mention | Other
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	2,500	3,361
Commercial loans | Special mention | Commercial real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	46	758
Commercial loans | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	19,159	3,032
Commercial loans | Substandard | Other
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	1,668	1,433
Commercial loans | Substandard | Commercial real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	17,467	1,559
Total SBA, SBA 504 and commercial loans
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	965,757	800,799
Total SBA, SBA 504 and commercial loans | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	942,745	790,275
Total SBA, SBA 504 and commercial loans | Special mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	2,546	7,042
Total SBA, SBA 504 and commercial loans | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	20,466	3,482
Residential mortgage loans
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	469,987	467,706
Residential mortgage loans | Performing
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	463,795	463,770
Residential mortgage loans | Nonperforming
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	6,192	3,936
Consumer loans
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	146,161	143,524
Consumer loans | Other
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	78,569	73,935
Consumer loans | Home equity
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	67,592	69,589
Consumer loans | Performing
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	145,656	143,504
Consumer loans | Performing | Other
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	78,569	73,915
Consumer loans | Performing | Home equity
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	67,087	69,589
Consumer loans | Nonperforming
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	505	20
Consumer loans | Nonperforming | Other
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment		20
Consumer loans | Nonperforming | Home equity
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	505
Total residential mortgage and consumer loans
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	616,148	611,230
Total residential mortgage and consumer loans | Performing
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	609,451	607,274
Total residential mortgage and consumer loans | Nonperforming
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans held for investment	$ 6,697	$ 3,956