Loans and Credit Quality - Loan Portfolio Segmented by Risk Class (Details) - USD ($) $ in Thousands	Mar. 31, 2017	Dec. 31, 2016
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	$ 965,225	$ 979,449
Less: Unearned origination fees, net of origination costs	(4,393)	(4,434)
Total loans	960,832	975,015
AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	912,007	939,573
AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	17,073	2,586
AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	36,145	37,290
Commercial
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	275,809	278,178
Commercial | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	243,330	257,639
Commercial | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	3,622	1,950
Commercial | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	28,857	18,589
Real estate construction one-to-four family
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	26,029	26,061
Real estate construction one-to-four family | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	26,029	26,061
Real estate construction one-to-four family | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Real estate construction one-to-four family | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Real estate construction other
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	67,975	72,159
Real estate construction other | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	67,975	72,159
Real estate construction other | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Real estate construction other | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Real estate term owner occupied
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	140,102	152,178
Real estate term owner occupied | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	133,657	135,359
Real estate term owner occupied | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	1,098	57
Real estate term owner occupied | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	5,347	16,762
Real estate term non-owner occupied
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	366,254	356,601
Real estate term non-owner occupied | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	354,217	355,903
Real estate term non-owner occupied | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	11,334	0
Real estate term non-owner occupied | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	703	698
Real estate term other
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	42,218	45,402
Real estate term other | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	41,560	44,733
Real estate term other | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Real estate term other | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	658	669
Consumer secured by 1st deeds of trust
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	22,762	23,589
Consumer secured by 1st deeds of trust | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	21,292	22,568
Consumer secured by 1st deeds of trust | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	968	501
Consumer secured by 1st deeds of trust | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	502	520
Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	24,076	25,281
Consumer other | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	23,947	25,151
Consumer other | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	51	78
Consumer other | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	$ 78	$ 52