Allowance for Loan Losses and Reserve for Unfunded Loan Commitments - Allowance for Credit Losses on Basis of Impairment Method (Details) - USD ($) $ in Thousands	Mar. 31, 2020	Dec. 31, 2019	Mar. 31, 2019	Dec. 31, 2018
Financing Receivable, Allowance for Credit Loss [Line Items]
Total	$ 17,376	$ 16,395	$ 15,684	$ 15,488
Total loans	1,428,919	1,412,029
SBA loans held for investment
Financing Receivable, Allowance for Credit Loss [Line Items]
Individually evaluated for impairment	138	41
Collectively evaluated for impairment	867	1,038
Total	1,005	1,079	1,678	1,655
Individually evaluated for impairment	1,200	1,105
Collectively evaluated for impairment	30,497	34,662
Total loans	31,697	35,767
Total commercial loans
Financing Receivable, Allowance for Credit Loss [Line Items]
Individually evaluated for impairment	46	373
Collectively evaluated for impairment	10,083	9,349
Total	10,129	9,722	8,793	8,705
Individually evaluated for impairment	1,307	1,234
Collectively evaluated for impairment	784,770	763,798
Total loans	786,077	765,032
Residential mortgage loans
Financing Receivable, Allowance for Credit Loss [Line Items]
Individually evaluated for impairment	0	0
Collectively evaluated for impairment	4,763	4,254
Total	4,763	4,254	4,016	3,900
Individually evaluated for impairment	0	0
Collectively evaluated for impairment	456,072	467,706
Total loans	456,072	467,706
Total consumer loans
Financing Receivable, Allowance for Credit Loss [Line Items]
Individually evaluated for impairment	0	0
Collectively evaluated for impairment	1,479	1,340
Total	1,479	1,340	$ 1,197	$ 1,228
Individually evaluated for impairment	0	0
Collectively evaluated for impairment	149,695	143,524
Total loans	149,695	143,524
Total
Financing Receivable, Allowance for Credit Loss [Line Items]
Individually evaluated for impairment	184	414
Collectively evaluated for impairment	17,192	15,981
Total	17,376	16,395
Individually evaluated for impairment	2,507	2,339
Collectively evaluated for impairment	1,421,034	1,409,690
Total loans	$ 1,423,541	$ 1,412,029