Loans and Credit Quality - Loan Portfolio Segmented By Risk Class (Details) - USD ($) $ in Thousands	Jun. 30, 2016	Dec. 31, 2015
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	$ 971,666	$ 985,399
Less: Unearned origination fees, net of origination costs	(4,320)	(4,612)
Total loans	967,346	980,787
Asset Quality Rating - Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	923,797	951,901
Asset Quality Rating - Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	4,808	808
Asset Quality Rating - Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	43,061	32,690
Commercial
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	337,354	329,534
Commercial | Asset Quality Rating - Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	314,597	313,689
Commercial | Asset Quality Rating - Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	3,109	536
Commercial | Asset Quality Rating - Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	19,648	15,309
Real estate construction one-to-four family
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	38,783	44,488
Real estate construction one-to-four family | Asset Quality Rating - Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	34,811	44,488
Real estate construction one-to-four family | Asset Quality Rating - Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Real estate construction one-to-four family | Asset Quality Rating - Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	3,972	0
Real estate construction other
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	57,453	74,931
Real estate construction other | Asset Quality Rating - Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	55,541	74,931
Real estate construction other | Asset Quality Rating - Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Real estate construction other | Asset Quality Rating - Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	1,912	0
Real estate term owner occupied
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	153,264	128,763
Real estate term owner occupied | Asset Quality Rating - Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	135,670	112,248
Real estate term owner occupied | Asset Quality Rating - Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	1,256	0
Real estate term owner occupied | Asset Quality Rating - Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	16,338	16,515
Real estate term non-owner occupied
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	291,600	314,069
Real estate term non-owner occupied | Asset Quality Rating - Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	291,379	313,710
Real estate term non-owner occupied | Asset Quality Rating - Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Real estate term non-owner occupied | Asset Quality Rating - Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	221	359
Real estate term other
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	39,078	38,029
Real estate term other | Asset Quality Rating - Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	38,793	37,938
Real estate term other | Asset Quality Rating - Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	285	91
Real estate term other | Asset Quality Rating - Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Consumer secured by 1st deeds of trust
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	27,031	26,673
Consumer secured by 1st deeds of trust | Asset Quality Rating - Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	25,970	26,015
Consumer secured by 1st deeds of trust | Asset Quality Rating - Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	152	171
Consumer secured by 1st deeds of trust | Asset Quality Rating - Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	909	487
Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	27,103	28,912
Consumer other | Asset Quality Rating - Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	27,036	28,882
Consumer other | Asset Quality Rating - Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	6	10
Consumer other | Asset Quality Rating - Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	$ 61	$ 20