Loans and Credit Quality - Past Due Loans and Nonaccrual Loans (Details) - USD ($) $ in Thousands	Jun. 30, 2018	Dec. 31, 2017
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	$ 1,855	$ 2,388
Nonaccrual	16,635	21,626
Current	953,419	935,095
Total financing receivables	971,909	959,109
Less: Unearned origination fees, net of origination costs	(4,207)	(4,156)
Total loans	967,702	954,953
30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	1,566	2,083
Nonaccrual	0	810
60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	289	53
Nonaccrual	41	0
Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	252
Nonaccrual	1,789	3,030
Commercial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	1,083	743
Nonaccrual	15,234	19,917
Current	311,416	292,854
Total financing receivables	327,733	313,514
Commercial | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	973	503
Nonaccrual	0	810
Commercial | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	110	0
Nonaccrual	0	0
Commercial | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	240
Nonaccrual	1,769	2,652
Real estate construction one-to-four family
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Current	31,064	31,201
Total financing receivables	31,064	31,201
Real estate construction one-to-four family | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate construction one-to-four family | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate construction one-to-four family | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate construction other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	90
Nonaccrual	0	0
Current	58,369	80,003
Total financing receivables	58,369	80,093
Real estate construction other | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	90
Real estate construction other | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate construction other | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate term owner occupied
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	966
Nonaccrual	1,331	1,331
Current	126,053	129,745
Total financing receivables	127,384	132,042
Real estate term owner occupied | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	966
Nonaccrual	0	0
Real estate term owner occupied | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Real estate term owner occupied | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Real estate term non-owner occupied
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Current	330,217	319,313
Total financing receivables	330,217	319,313
Real estate term non-owner occupied | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate term non-owner occupied | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate term non-owner occupied | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate term other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	611	0
Nonaccrual	0	0
Current	54,820	40,411
Total financing receivables	55,431	40,411
Real estate term other | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	584	0
Real estate term other | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	27	0
Real estate term other | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Consumer secured by 1st deeds of trust
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	152	363
Nonaccrual	0	378
Current	20,668	21,875
Total financing receivables	20,820	22,616
Consumer secured by 1st deeds of trust | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	363
Nonaccrual		0
Consumer secured by 1st deeds of trust | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	152	0
Nonaccrual		0
Consumer secured by 1st deeds of trust | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual		378
Consumer other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	9	226
Nonaccrual	70	0
Current	20,812	19,693
Total financing receivables	20,891	19,919
Consumer other | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	9	161
Nonaccrual	0
Consumer other | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	53
Nonaccrual	41
Consumer other | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	$ 12
Nonaccrual	$ 20