Loans and Credit Quality - Past Due Loans and Nonaccrual Loans (Details) - USD ($) $ in Thousands	Sep. 30, 2018	Dec. 31, 2017
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	$ 2,808	$ 2,388
Nonaccrual	16,728	21,626
Current	966,826	935,095
Total financing receivables	986,362	959,109
Less: Unearned origination fees, net of origination costs	(4,355)	(4,156)
Total loans	982,007	954,953
30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	1,331	2,083
Nonaccrual	121	810
60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	1,325	53
Nonaccrual	600	0
Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	152	252
Nonaccrual	4,728	3,030
Commercial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	1,098	743
Nonaccrual	14,734	19,917
Current	317,300	292,854
Total financing receivables	333,132	313,514
Commercial | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	1,098	503
Nonaccrual	121	810
Commercial | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	600	0
Commercial | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	240
Nonaccrual	2,742	2,652
Real estate construction one-to-four family
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Current	36,119	31,201
Total financing receivables	36,119	31,201
Real estate construction one-to-four family | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate construction one-to-four family | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate construction one-to-four family | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate construction other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	90
Nonaccrual	0	0
Current	61,857	80,003
Total financing receivables	61,857	80,093
Real estate construction other | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	90
Real estate construction other | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate construction other | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate term owner occupied
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	949	966
Nonaccrual	1,694	1,331
Current	127,523	129,745
Total financing receivables	130,166	132,042
Real estate term owner occupied | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	211	966
Nonaccrual	0	0
Real estate term owner occupied | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	738	0
Nonaccrual	0	0
Real estate term owner occupied | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	1,694	0
Real estate term non-owner occupied
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Current	340,820	319,313
Total financing receivables	340,820	319,313
Real estate term non-owner occupied | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate term non-owner occupied | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate term non-owner occupied | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate term other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	577	0
Nonaccrual	26	0
Current	40,890	40,411
Total financing receivables	41,493	40,411
Real estate term other | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0
Real estate term other | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	577	0
Nonaccrual	0
Real estate term other | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	26
Consumer secured by 1st deeds of trust
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	152	363
Nonaccrual	226	378
Current	19,824	21,875
Total financing receivables	20,202	22,616
Consumer secured by 1st deeds of trust | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	363
Nonaccrual	0	0
Consumer secured by 1st deeds of trust | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Consumer secured by 1st deeds of trust | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	152	0
Nonaccrual	226	378
Consumer other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	32	226
Nonaccrual	48	0
Current	22,493	19,693
Total financing receivables	22,573	19,919
Consumer other | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	22	161
Nonaccrual	0
Consumer other | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	10	53
Nonaccrual	0
Consumer other | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	$ 12
Nonaccrual	$ 40