Loans and Credit Quality - Past Due Loans and Nonaccrual Loans (Details) - USD ($) $ in Thousands	Jun. 30, 2017	Dec. 31, 2016
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	$ 3,220	$ 2,643
Nonaccrual	22,899	13,893
Current	969,234	962,913
Total financing receivables	995,353	979,449
Less: Unearned origination fees, net of origination costs	(4,144)	(4,434)
Total loans	991,209	975,015
30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	2,530	1,448
Nonaccrual	0	8,750
60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	222	739
Nonaccrual	6,986	0
Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	468	456
Nonaccrual	3,661	4,375
Commercial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	747	545
Nonaccrual	22,778	13,500
Current	285,968	264,133
Total financing receivables	309,493	278,178
Commercial | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	540	0
Nonaccrual	0	8,750
Commercial | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	103	141
Nonaccrual	6,986	0
Commercial | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	104	404
Nonaccrual	3,635	4,208
Real estate construction one-to-four family
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Current	24,714	26,061
Total financing receivables	24,714	26,061
Real estate construction one-to-four family | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate construction one-to-four family | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate construction one-to-four family | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate construction other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Current	73,999	72,159
Total financing receivables	73,999	72,159
Real estate construction other | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate construction other | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate construction other | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate term owner occupied
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	1,331	887
Nonaccrual	0	29
Current	138,144	151,262
Total financing receivables	139,475	152,178
Real estate term owner occupied | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	1,331	887
Nonaccrual		0
Real estate term owner occupied | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual		0
Real estate term owner occupied | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual		0
Real estate term non-owner occupied
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	95	197
Current	359,368	356,404
Total financing receivables	359,463	356,601
Real estate term non-owner occupied | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Real estate term non-owner occupied | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Real estate term non-owner occupied | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Real estate term other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	613	0
Nonaccrual	0	0
Current	41,586	45,402
Total financing receivables	42,199	45,402
Real estate term other | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	613	0
Real estate term other | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate term other | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Consumer secured by 1st deeds of trust
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	461	908
Nonaccrual	0	167
Current	22,952	22,514
Total financing receivables	23,413	23,589
Consumer secured by 1st deeds of trust | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	390
Nonaccrual		0
Consumer secured by 1st deeds of trust | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	119	518
Nonaccrual		0
Consumer secured by 1st deeds of trust | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	342	0
Nonaccrual		167
Consumer other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	68	303
Nonaccrual	26	0
Current	22,503	24,978
Total financing receivables	22,597	25,281
Consumer other | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	46	171
Nonaccrual	0
Consumer other | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	80
Nonaccrual	0
Consumer other | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	22	$ 52
Nonaccrual	$ 26