Loans and Credit Quality - Loan Portfolio Segmented By Risk Class (Details) - USD ($) $ in Thousands	Dec. 31, 2017	Dec. 31, 2016
Financing Receivable, Recorded Investment [Line Items]
Subtotal	$ 959,823	$ 979,449
Less: Unearned origination fees, net of origination costs	(4,156)	(4,434)
Total loans	955,667	975,015
Asset Quality Rating - Pass
Financing Receivable, Recorded Investment [Line Items]
Subtotal	905,736	939,573
Asset Quality Rating - Special Mention
Financing Receivable, Recorded Investment [Line Items]
Subtotal	18,726	2,586
Asset Quality Rating - Substandard
Financing Receivable, Recorded Investment [Line Items]
Subtotal	35,361	37,290
Commercial
Financing Receivable, Recorded Investment [Line Items]
Subtotal	313,769	278,178
Commercial | Asset Quality Rating - Pass
Financing Receivable, Recorded Investment [Line Items]
Subtotal	277,626	257,639
Commercial | Asset Quality Rating - Special Mention
Financing Receivable, Recorded Investment [Line Items]
Subtotal	4,921	1,950
Commercial | Asset Quality Rating - Substandard
Financing Receivable, Recorded Investment [Line Items]
Subtotal	31,222	18,589
Real estate construction one-to-four family
Financing Receivable, Recorded Investment [Line Items]
Subtotal	31,201	26,061
Real estate construction one-to-four family | Asset Quality Rating - Pass
Financing Receivable, Recorded Investment [Line Items]
Subtotal	31,201	26,061
Real estate construction one-to-four family | Asset Quality Rating - Special Mention
Financing Receivable, Recorded Investment [Line Items]
Subtotal	0	0
Real estate construction one-to-four family | Asset Quality Rating - Substandard
Financing Receivable, Recorded Investment [Line Items]
Subtotal	0	0
Real estate construction other
Financing Receivable, Recorded Investment [Line Items]
Subtotal	80,093	72,159
Real estate construction other | Asset Quality Rating - Pass
Financing Receivable, Recorded Investment [Line Items]
Subtotal	80,093	72,159
Real estate construction other | Asset Quality Rating - Special Mention
Financing Receivable, Recorded Investment [Line Items]
Subtotal	0	0
Real estate construction other | Asset Quality Rating - Substandard
Financing Receivable, Recorded Investment [Line Items]
Subtotal	0	0
Real estate term owner occupied
Financing Receivable, Recorded Investment [Line Items]
Subtotal	132,098	152,178
Real estate term owner occupied | Asset Quality Rating - Pass
Financing Receivable, Recorded Investment [Line Items]
Subtotal	127,115	135,359
Real estate term owner occupied | Asset Quality Rating - Special Mention
Financing Receivable, Recorded Investment [Line Items]
Subtotal	2,095	57
Real estate term owner occupied | Asset Quality Rating - Substandard
Financing Receivable, Recorded Investment [Line Items]
Subtotal	2,888	16,762
Real estate term non-owner occupied
Financing Receivable, Recorded Investment [Line Items]
Subtotal	319,459	356,601
Real estate term non-owner occupied | Asset Quality Rating - Pass
Financing Receivable, Recorded Investment [Line Items]
Subtotal	307,926	355,903
Real estate term non-owner occupied | Asset Quality Rating - Special Mention
Financing Receivable, Recorded Investment [Line Items]
Subtotal	11,051	0
Real estate term non-owner occupied | Asset Quality Rating - Substandard
Financing Receivable, Recorded Investment [Line Items]
Subtotal	482	698
Real estate term other
Financing Receivable, Recorded Investment [Line Items]
Subtotal	40,411	45,402
Real estate term other | Asset Quality Rating - Pass
Financing Receivable, Recorded Investment [Line Items]
Subtotal	39,777	44,733
Real estate term other | Asset Quality Rating - Special Mention
Financing Receivable, Recorded Investment [Line Items]
Subtotal	634	0
Real estate term other | Asset Quality Rating - Substandard
Financing Receivable, Recorded Investment [Line Items]
Subtotal	0	669
Consumer secured by 1st deeds of trust
Financing Receivable, Recorded Investment [Line Items]
Subtotal	22,873	23,589
Consumer secured by 1st deeds of trust | Asset Quality Rating - Pass
Financing Receivable, Recorded Investment [Line Items]
Subtotal	22,103	22,568
Consumer secured by 1st deeds of trust | Asset Quality Rating - Special Mention
Financing Receivable, Recorded Investment [Line Items]
Subtotal	3	501
Consumer secured by 1st deeds of trust | Asset Quality Rating - Substandard
Financing Receivable, Recorded Investment [Line Items]
Subtotal	767	520
Consumer other
Financing Receivable, Recorded Investment [Line Items]
Subtotal	19,919	25,281
Consumer other | Asset Quality Rating - Pass
Financing Receivable, Recorded Investment [Line Items]
Subtotal	19,895	25,151
Consumer other | Asset Quality Rating - Special Mention
Financing Receivable, Recorded Investment [Line Items]
Subtotal	22	78
Consumer other | Asset Quality Rating - Substandard
Financing Receivable, Recorded Investment [Line Items]
Subtotal	$ 2	$ 52