Loans and Credit Quality - Past Due Loans and Nonaccrual Loans (Details) - USD ($) $ in Thousands	Sep. 30, 2017	Dec. 31, 2016
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	$ 7,254	$ 2,643
Nonaccrual	24,317	13,893
Current	961,657	962,913
Total financing receivables	993,228	979,449
Less: Unearned origination fees, net of origination costs	(3,975)	(4,434)
Total loans	989,253	975,015
30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	6,782	1,448
Nonaccrual	0	8,750
60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	258	739
Nonaccrual	0	0
Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	214	456
Nonaccrual	6,590	4,375
Commercial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	2,112	545
Nonaccrual	22,580	13,500
Current	290,820	264,133
Total financing receivables	315,512	278,178
Commercial | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	1,640	0
Nonaccrual	0	8,750
Commercial | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	258	141
Nonaccrual	0	0
Commercial | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	214	404
Nonaccrual	4,853	4,208
Real estate construction one-to-four family
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Current	27,543	26,061
Total financing receivables	27,543	26,061
Real estate construction one-to-four family | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate construction one-to-four family | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate construction one-to-four family | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate construction other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	367	0
Nonaccrual	0	0
Current	83,517	72,159
Total financing receivables	83,884	72,159
Real estate construction other | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	367	0
Real estate construction other | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate construction other | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate term owner occupied
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	4,426	887
Nonaccrual	1,331	29
Current	129,296	151,262
Total financing receivables	135,053	152,178
Real estate term owner occupied | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	4,426	887
Nonaccrual	0	0
Real estate term owner occupied | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Real estate term owner occupied | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	1,331	0
Real estate term non-owner occupied
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	197
Current	344,968	356,404
Total financing receivables	344,968	356,601
Real estate term non-owner occupied | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual		0
Real estate term non-owner occupied | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual		0
Real estate term non-owner occupied | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual		0
Real estate term other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Current	41,326	45,402
Total financing receivables	41,326	45,402
Real estate term other | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate term other | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate term other | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Consumer secured by 1st deeds of trust
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	4	908
Nonaccrual	380	167
Current	22,949	22,514
Total financing receivables	23,333	23,589
Consumer secured by 1st deeds of trust | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	4	390
Nonaccrual	0	0
Consumer secured by 1st deeds of trust | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	518
Nonaccrual	0	0
Consumer secured by 1st deeds of trust | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	380	167
Consumer other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	345	303
Nonaccrual	26	0
Current	21,238	24,978
Total financing receivables	21,609	25,281
Consumer other | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	345	171
Nonaccrual	0
Consumer other | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	80
Nonaccrual	0
Consumer other | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	$ 52
Nonaccrual	$ 26