Loans and Credit Quality - Past Due Loans And Nonaccrual Loans (Details) - USD ($) $ in Thousands	Dec. 31, 2017	Dec. 31, 2016
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	$ 2,388	$ 2,643
Nonaccrual loans	21,626	13,893
Current loans	935,809	962,913
Total financing receivables	959,823	979,449
Less: Unearned origination fees, net of origination costs	(4,156)	(4,434)
Total loans	955,667	975,015
30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	2,083	1,448
Nonaccrual loans	810	8,750
60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	53	739
Nonaccrual loans	0	0
Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	252	456
Nonaccrual loans	3,030	4,375
Commercial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	743	545
Nonaccrual loans	19,917	13,500
Current loans	293,109	264,133
Total financing receivables	313,769	278,178
Commercial | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	503	0
Nonaccrual loans	810	8,750
Commercial | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	141
Nonaccrual loans	0	0
Commercial | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	240	404
Nonaccrual loans	2,652	4,208
Real estate construction one-to-four family
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Nonaccrual loans	0	0
Current loans	31,201	26,061
Total financing receivables	31,201	26,061
Real estate construction one-to-four family | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Real estate construction one-to-four family | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Real estate construction one-to-four family | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Real estate construction other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	90	0
Nonaccrual loans	0	0
Current loans	80,003	72,159
Total financing receivables	80,093	72,159
Real estate construction other | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	90	0
Real estate construction other | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Real estate construction other | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Real estate term owner occupied
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	966	887
Nonaccrual loans	1,331	29
Current loans	129,801	151,262
Total financing receivables	132,098	152,178
Real estate term owner occupied | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	966	887
Nonaccrual loans	0	0
Real estate term owner occupied | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Nonaccrual loans	0	0
Real estate term owner occupied | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Nonaccrual loans	0	0
Real estate term non-owner occupied
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Nonaccrual loans	0	197
Current loans	319,459	356,404
Total financing receivables	319,459	356,601
Real estate term non-owner occupied | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Nonaccrual loans		0
Real estate term non-owner occupied | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Nonaccrual loans		0
Real estate term non-owner occupied | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Nonaccrual loans		0
Real estate term other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Nonaccrual loans	0	0
Current loans	40,411	45,402
Total financing receivables	40,411	45,402
Real estate term other | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Real estate term other | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Real estate term other | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Consumer secured by 1st deeds of trust
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	363	908
Nonaccrual loans	378	167
Current loans	22,132	22,514
Total financing receivables	22,873	23,589
Consumer secured by 1st deeds of trust | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	363	390
Nonaccrual loans	0	0
Consumer secured by 1st deeds of trust | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	518
Nonaccrual loans	0	0
Consumer secured by 1st deeds of trust | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Nonaccrual loans	378	167
Consumer other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	226	303
Nonaccrual loans	0	0
Current loans	19,693	24,978
Total financing receivables	19,919	25,281
Consumer other | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	161	171
Consumer other | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	53	80
Consumer other | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	$ 12	$ 52