Loans and Credit Quality - Past Due Loans and Nonaccrual Loans (Details) - USD ($) $ in Thousands	Mar. 31, 2020	Dec. 31, 2019
Financing Receivable, Past Due [Line Items]
Total Past Due	$ 3,554	$ 1,534
Nonaccrual	15,074	15,356
Current	1,068,182	1,031,566
Total financing receivable	1,086,810	1,048,456
Less: Unearned origination fees, net of origination costs	(4,937)	(5,085)
Total loans	1,081,873	1,043,371
30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	3,317	1,534
Nonaccrual	5,344	1,937
60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	237	0
Nonaccrual	945	474
Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	6,735	6,010
Commercial
Financing Receivable, Past Due [Line Items]
Total Past Due	798	270
Nonaccrual	9,340	9,153
Current	424,694	403,267
Total financing receivable	434,832	412,690
Total loans	433,388	411,327
Commercial | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	561	270
Nonaccrual	4,285	270
Commercial | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	237	0
Nonaccrual	308	385
Commercial | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	3,030	2,862
Real estate construction one-to-four family
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	915	1,349
Current	35,427	37,469
Total financing receivable	36,342	38,818
Total loans	36,130	38,503
Real estate construction one-to-four family | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Real estate construction one-to-four family | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Real estate construction one-to-four family | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	915	1,349
Real estate construction other
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Current	73,507	61,808
Total financing receivable	73,507	61,808
Total loans	72,725	60,906
Real estate construction other | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Real estate construction other | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Real estate construction other | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Real estate term owner occupied
Financing Receivable, Past Due [Line Items]
Total Past Due	2,502	338
Nonaccrual	3,459	3,489
Current	140,492	135,064
Total financing receivable	146,453	138,891
Total loans	145,693	138,181
Real estate term owner occupied | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	2,502	338
Nonaccrual	1,029	1,641
Real estate term owner occupied | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	637	0
Real estate term owner occupied | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	1,614	623
Real estate term non-owner occupied
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Current	313,472	312,960
Total financing receivable	313,472	312,960
Total loans	311,869	311,302
Real estate term non-owner occupied | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Real estate term non-owner occupied | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Real estate term non-owner occupied | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Real estate term other
Financing Receivable, Past Due [Line Items]
Total Past Due	0	26
Nonaccrual	1,176	1,176
Current	41,105	41,304
Total financing receivable	42,281	42,506
Total loans	41,980	42,200
Real estate term other | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	26
Nonaccrual	0	0
Real estate term other | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Real estate term other | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	1,176	1,176
Consumer secured by 1st deeds of trust
Financing Receivable, Past Due [Line Items]
Total Past Due	254	750
Nonaccrual	67	68
Current	15,253	15,380
Total financing receivable	15,574	16,198
Total loans	15,562	16,191
Consumer secured by 1st deeds of trust | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	254	750
Nonaccrual	0	0
Consumer secured by 1st deeds of trust | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Consumer secured by 1st deeds of trust | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Consumer other
Financing Receivable, Past Due [Line Items]
Total Past Due	0	150
Nonaccrual	117	121
Current	24,232	24,314
Total financing receivable	24,349	24,585
Total loans	24,526	24,761
Consumer other | 30-59 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	150
Nonaccrual	30	26
Consumer other | 60-89 Days Past Due Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	89
Consumer other | Greater Than 90 Days Still Accruing
Financing Receivable, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	$ 0	$ 0