Loans and Credit Quality - Past Due Loans And Nonaccrual Loans (Details) - USD ($) $ in Thousands	Dec. 31, 2016	Dec. 31, 2015
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	$ 2,643	$ 1,150
Nonaccrual loans	13,893	3,686
Current loans	962,913	980,563
Total financing receivables	979,449	985,399
Less: Unearned origination fees, net of origination costs	(4,434)	(4,612)
Total loans	975,015	980,787
30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	1,448	1,129
Nonaccrual loans	8,750	0
60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	739	21
Nonaccrual loans	0	82
Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	456	0
Nonaccrual loans	4,375	418
Commercial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	545	263
Nonaccrual loans	13,500	3,013
Current loans	264,133	269,165
Total financing receivables	278,178	272,441
Commercial | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	242
Nonaccrual loans	8,750	0
Commercial | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	141	21
Nonaccrual loans	0	82
Commercial | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	404	0
Nonaccrual loans	4,208	290
Real estate construction one-to-four family
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Nonaccrual loans	0	0
Current loans	26,061	44,488
Total financing receivables	26,061	44,488
Real estate construction one-to-four family | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Real estate construction one-to-four family | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Real estate construction one-to-four family | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Real estate construction other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Nonaccrual loans	0	0
Current loans	72,159	74,956
Total financing receivables	72,159	74,956
Real estate construction other | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Real estate construction other | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Real estate construction other | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Real estate term owner occupied
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	887	0
Nonaccrual loans	29	38
Current loans	151,262	143,703
Total financing receivables	152,178	143,741
Real estate term owner occupied | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	887	0
Nonaccrual loans	0	0
Real estate term owner occupied | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Nonaccrual loans	0	0
Real estate term owner occupied | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Nonaccrual loans	0	0
Real estate term non-owner occupied
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Nonaccrual loans	197	359
Current loans	356,404	347,157
Total financing receivables	356,601	347,516
Real estate term non-owner occupied | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Nonaccrual loans	0	0
Real estate term non-owner occupied | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Nonaccrual loans	0	0
Real estate term non-owner occupied | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Nonaccrual loans	0	108
Real estate term other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	289
Nonaccrual loans	0	0
Current loans	45,402	46,383
Total financing receivables	45,402	46,672
Real estate term other | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	289
Real estate term other | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Real estate term other | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Consumer secured by 1st deeds of trust
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	908	568
Nonaccrual loans	167	256
Current loans	22,514	25,849
Total financing receivables	23,589	26,673
Consumer secured by 1st deeds of trust | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	390	568
Nonaccrual loans	0	0
Consumer secured by 1st deeds of trust | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	518	0
Nonaccrual loans	0	0
Consumer secured by 1st deeds of trust | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Nonaccrual loans	167	0
Consumer other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	303	30
Nonaccrual loans	0	20
Current loans	24,978	28,862
Total financing receivables	25,281	28,912
Consumer other | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	171	30
Nonaccrual loans		0
Consumer other | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	80	0
Nonaccrual loans		0
Consumer other | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	$ 52	0
Nonaccrual loans		$ 20