Loans and Credit Quality - Past Due Loans And Nonaccrual Loans (Details) - USD ($) $ in Thousands	Jun. 30, 2016	Dec. 31, 2015
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	$ 1,211	$ 1,150
Nonaccrual loans	11,184	3,686
Total current, subtotal	959,271	980,563
Total financing receivables	971,666	985,399
Less: Unearned origination fees, net of origination costs	(4,320)	(4,612)
Total loans	967,346	980,787
30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	1,027	1,129
60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	137	21
Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	47	0
Commercial
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	471	263
Nonaccrual loans	4,479	3,013
Total current, subtotal	332,404	326,258
Total financing receivables	337,354	329,534
Commercial | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	334	242
Commercial | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	137	21
Commercial | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Real estate construction one-to-four family
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Nonaccrual loans	3,972	0
Total current, subtotal	34,811	44,488
Total financing receivables	38,783	44,488
Real estate construction one-to-four family | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Real estate construction one-to-four family | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Real estate construction one-to-four family | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Real estate construction other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Nonaccrual loans	1,912	0
Total current, subtotal	55,541	74,931
Total financing receivables	57,453	74,931
Real estate construction other | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Real estate construction other | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Real estate construction other | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Real estate term owner occupied
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	550	0
Nonaccrual loans	34	38
Total current, subtotal	152,680	128,725
Total financing receivables	153,264	128,763
Real estate term owner occupied | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	550	0
Real estate term owner occupied | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Real estate term owner occupied | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Real estate term non-owner occupied
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Nonaccrual loans	221	359
Total current, subtotal	291,379	313,710
Total financing receivables	291,600	314,069
Real estate term non-owner occupied | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Real estate term non-owner occupied | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Real estate term non-owner occupied | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Real estate term other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	289
Nonaccrual loans	0	0
Total current, subtotal	39,078	37,740
Total financing receivables	39,078	38,029
Real estate term other | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	289
Real estate term other | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Real estate term other | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Consumer secured by 1st deeds of trust
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	143	568
Nonaccrual loans	552	256
Total current, subtotal	26,336	25,849
Total financing receivables	27,031	26,673
Consumer secured by 1st deeds of trust | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	143	568
Consumer secured by 1st deeds of trust | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Consumer secured by 1st deeds of trust | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Consumer other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	47	30
Nonaccrual loans	14	20
Total current, subtotal	27,042	28,862
Total financing receivables	27,103	28,912
Consumer other | 30-59 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	30
Consumer other | 60-89 Days Past Due Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	0	0
Consumer other | Greater Than 90 Days Still Accruing
Financing Receivable, Recorded Investment, Past Due [Line Items]
Past due loans	$ 47	$ 0