Loans and Credit Quality - Past Due Loans and Nonaccrual Loans (Details) - USD ($) $ in Thousands	Mar. 31, 2018	Dec. 31, 2017
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	$ 8,968	$ 2,388
Nonaccrual	18,895	21,626
Current	943,820	935,095
Total financing receivables	971,683	959,109
Less: Unearned origination fees, net of origination costs	(4,108)	(4,156)
Total loans	967,575	954,953
30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	8,597	2,083
Nonaccrual	15	810
60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	287	53
Nonaccrual	0	0
Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	84	252
Nonaccrual	3,560	3,030
Commercial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	5,729	743
Nonaccrual	17,268	19,917
Current	293,084	292,854
Total financing receivables	316,081	313,514
Commercial | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	5,506	503
Nonaccrual	15	810
Commercial | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	223	0
Nonaccrual	0	0
Commercial | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	240
Nonaccrual	1,933	2,652
Real estate construction one-to-four family
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Current	30,310	31,201
Total financing receivables	30,310	31,201
Real estate construction one-to-four family | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate construction one-to-four family | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate construction one-to-four family | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate construction other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	90
Nonaccrual	0	0
Current	54,947	80,003
Total financing receivables	54,947	80,093
Real estate construction other | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	90
Real estate construction other | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate construction other | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate term owner occupied
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	1,914	966
Nonaccrual	1,331	1,331
Current	129,344	129,745
Total financing receivables	132,589	132,042
Real estate term owner occupied | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	1,914	966
Nonaccrual	0	0
Real estate term owner occupied | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Real estate term owner occupied | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	1,331	0
Real estate term non-owner occupied
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	645	0
Nonaccrual	0	0
Current	338,976	319,313
Total financing receivables	339,621	319,313
Real estate term non-owner occupied | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	645	0
Real estate term non-owner occupied | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate term non-owner occupied | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate term other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Current	56,294	40,411
Total financing receivables	56,294	40,411
Real estate term other | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate term other | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Real estate term other | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Consumer secured by 1st deeds of trust
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	468	363
Nonaccrual	285	378
Current	21,143	21,875
Total financing receivables	21,896	22,616
Consumer secured by 1st deeds of trust | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	468	363
Nonaccrual	0	0
Consumer secured by 1st deeds of trust | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	0	0
Consumer secured by 1st deeds of trust | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	0	0
Nonaccrual	285	378
Consumer other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	212	226
Nonaccrual	11	0
Current	19,722	19,693
Total financing receivables	19,945	19,919
Consumer other | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	64	161
Nonaccrual	0
Consumer other | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	64	53
Nonaccrual	0
Consumer other | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Total Past Due	84	$ 12
Nonaccrual	$ 11