Loans and Credit Quality - Loan Portfolio Segmented by Risk Class (Details) - USD ($) $ in Thousands	Jun. 30, 2017	Dec. 31, 2016
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	$ 995,353	$ 979,449
Less: Unearned origination fees, net of origination costs	(4,144)	(4,434)
Total loans	991,209	975,015
AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	941,760	939,573
AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	18,189	2,586
AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	35,404	37,290
Commercial
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	309,493	278,178
Commercial | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	276,298	257,639
Commercial | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	5,567	1,950
Commercial | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	27,628	18,589
Real estate construction one-to-four family
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	24,714	26,061
Real estate construction one-to-four family | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	24,714	26,061
Real estate construction one-to-four family | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Real estate construction one-to-four family | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Real estate construction other
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	73,999	72,159
Real estate construction other | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	73,999	72,159
Real estate construction other | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Real estate construction other | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Real estate term owner occupied
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	139,475	152,178
Real estate term owner occupied | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	133,036	135,359
Real estate term owner occupied | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	348	57
Real estate term owner occupied | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	6,091	16,762
Real estate term non-owner occupied
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	359,463	356,601
Real estate term non-owner occupied | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	347,489	355,903
Real estate term non-owner occupied | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	11,291	0
Real estate term non-owner occupied | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	683	698
Real estate term other
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	42,199	45,402
Real estate term other | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	41,555	44,733
Real estate term other | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	0	0
Real estate term other | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	644	669
Consumer secured by 1st deeds of trust
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	23,413	23,589
Consumer secured by 1st deeds of trust | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	22,143	22,568
Consumer secured by 1st deeds of trust | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	938	501
Consumer secured by 1st deeds of trust | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	332	520
Consumer other
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	22,597	25,281
Consumer other | AQR Pass
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	22,526	25,151
Consumer other | AQR Special Mention
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	45	78
Consumer other | AQR Substandard
Financing Receivable, Recorded Investment [Line Items]
Total financing receivables	$ 26	$ 52