Loans - Aging Analysis of Past Due and Nonaccrual Loans by Class (Details) - USD ($) $ in Thousands	Sep. 30, 2020	Dec. 31, 2019
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	$ 17,189	$ 17,517
Loans receivable, nonaccrual status	8,988	5,649
Loan receivable, current	1,596,102	1,408,041
Total loans held for investment	1,589,910	1,394,512
Total loans	1,607,099	1,412,029
SBA loans held for sale	6,192	13,529
Total loans	1,613,291	1,425,558
SBA 504 loans
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due		1,808
Loan receivable, current	21,811	24,918
Total loans	21,811	26,726
30-59 days past due
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	6,857	7,163
60-89 days past due
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	1,344	3,775
60-89 days past due | SBA 504 loans
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due		1,808
90 days and still accruing
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due		930
SBA loans held for investment
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	3,644	2,212
Loans receivable, nonaccrual status	3,414	1,164
Loan receivable, current	43,481	33,555
Total loans	47,125	35,767
SBA loans held for investment | 30-59 days past due
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	230	1,048
SBA PPP loans
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	32
Loans receivable, nonaccrual status	32
Loan receivable, current	138,863
Total loans	138,895	0
Commercial loans
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans	799,573	765,032
Commercial loans | Other
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	276	387
Loans receivable, nonaccrual status	130	316
Loan receivable, current	115,439	111,627
Total loans	115,715	112,014
Commercial loans | Commercial real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	2,318	428
Loans receivable, nonaccrual status	397	213
Loan receivable, current	593,925	578,215
Total loans	596,243	578,643
Commercial loans | Commercial real estate construction
Financing Receivable, Credit Quality Indicator [Line Items]
Loan receivable, current	65,804	47,649
Total loans	65,804	47,649
Commercial loans | 30-59 days past due | Other
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	146	71
Commercial loans | 30-59 days past due | Commercial real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	1,343	215
Commercial loans | 60-89 days past due | Commercial real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	578
Residential mortgage loans
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	7,722	10,925
Loans receivable, nonaccrual status	3,722	3,936
Loan receivable, current	465,698	456,781
Total loans	473,420	467,706
Residential mortgage loans | 30-59 days past due
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	3,424	4,383
Residential mortgage loans | 60-89 days past due
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	576	1,676
Residential mortgage loans | 90 days and still accruing
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due		930
Consumer loans
Financing Receivable, Credit Quality Indicator [Line Items]
Total loans	148,086	143,524
Consumer loans | Other
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	797	133
Loans receivable, nonaccrual status		20
Loan receivable, current	80,032	73,802
Total loans	80,829	73,935
Consumer loans | Home equity
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	2,400	1,624
Loans receivable, nonaccrual status	1,293
Loan receivable, current	64,857	67,965
Total loans	67,257	69,589
Consumer loans | 30-59 days past due | Other
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	797
Consumer loans | 30-59 days past due | Home equity
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	917	1,446
Consumer loans | 60-89 days past due | Other
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due		113
Consumer loans | 60-89 days past due | Home equity
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, past due	190	178
Collateral Pledged | Small Business Administration
Financing Receivable, Credit Quality Indicator [Line Items]
Loans receivable, nonaccrual status	$ 812	$ 59