Loans and Credit Quality - Loan Portfolio Segmented By Risk Class (Details) - USD ($) $ in Thousands	Sep. 30, 2016	Dec. 31, 2015
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	$ 1,001,484	$ 985,399
Less: Unearned origination fees, net of origination costs	(4,408)	(4,612)
Total loans	997,076	980,787
Asset Quality Rating - Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	956,098	951,901
Asset Quality Rating - Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	1,804	808
Asset Quality Rating - Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	43,582	32,690
Commercial
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	342,632	329,534
Commercial | Asset Quality Rating - Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	322,095	313,689
Commercial | Asset Quality Rating - Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	766	536
Commercial | Asset Quality Rating - Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	19,771	15,309
Real estate construction one-to-four family
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	37,002	44,488
Real estate construction one-to-four family | Asset Quality Rating - Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	33,030	44,488
Real estate construction one-to-four family | Asset Quality Rating - Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Real estate construction one-to-four family | Asset Quality Rating - Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	3,972	0
Real estate construction other
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	72,091	74,931
Real estate construction other | Asset Quality Rating - Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	70,179	74,931
Real estate construction other | Asset Quality Rating - Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Real estate construction other | Asset Quality Rating - Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	1,912	0
Real estate term owner occupied
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	166,941	128,763
Real estate term owner occupied | Asset Quality Rating - Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	149,409	112,248
Real estate term owner occupied | Asset Quality Rating - Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	693	0
Real estate term owner occupied | Asset Quality Rating - Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	16,839	16,515
Real estate term non-owner occupied
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	288,975	314,069
Real estate term non-owner occupied | Asset Quality Rating - Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	288,766	313,710
Real estate term non-owner occupied | Asset Quality Rating - Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Real estate term non-owner occupied | Asset Quality Rating - Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	209	359
Real estate term other
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	41,864	38,029
Real estate term other | Asset Quality Rating - Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	41,660	37,938
Real estate term other | Asset Quality Rating - Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	204	91
Real estate term other | Asset Quality Rating - Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Consumer secured by 1st deeds of trust
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	25,957	26,673
Consumer secured by 1st deeds of trust | Asset Quality Rating - Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	24,945	26,015
Consumer secured by 1st deeds of trust | Asset Quality Rating - Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	135	171
Consumer secured by 1st deeds of trust | Asset Quality Rating - Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	877	487
Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	26,022	28,912
Consumer other | Asset Quality Rating - Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	26,014	28,882
Consumer other | Asset Quality Rating - Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	6	10
Consumer other | Asset Quality Rating - Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	$ 2	$ 20