Loans and Credit Quality - Loan Portfolio Segmented by Risk Class (Details) - USD ($) $ in Thousands	Mar. 31, 2020	Dec. 31, 2019
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	$ 1,086,810	$ 1,048,456
Less: Unearned origination fees, net of origination costs	(4,937)	(5,085)
Total loans	1,081,873	1,043,371
AQR Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	1,036,817	994,583
AQR Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	26,044	29,660
AQR Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	23,834	24,213
AQR Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	46
AQR Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	69
Commercial
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	434,832	412,690
Total loans	433,388	411,327
Commercial | AQR Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	417,970	394,107
Commercial | AQR Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	2,117	2,279
Commercial | AQR Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	14,699	16,304
Commercial | AQR Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	46
Commercial | AQR Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0
Real estate construction one-to-four family
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	36,342	38,818
Total loans	36,130	38,503
Real estate construction one-to-four family | AQR Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	33,920	34,132
Real estate construction one-to-four family | AQR Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	1,507	3,337
Real estate construction one-to-four family | AQR Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	915	1,349
Real estate construction one-to-four family | AQR Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0
Real estate construction one-to-four family | AQR Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0
Real estate construction other
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	73,507	61,808
Total loans	72,725	60,906
Real estate construction other | AQR Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	73,507	61,808
Real estate construction other | AQR Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0	0
Real estate construction other | AQR Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0	0
Real estate construction other | AQR Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0
Real estate construction other | AQR Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0
Real estate term owner occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	146,453	138,891
Total loans	145,693	138,181
Real estate term owner occupied | AQR Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	135,880	129,959
Real estate term owner occupied | AQR Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	3,733	3,828
Real estate term owner occupied | AQR Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	6,840	5,104
Real estate term owner occupied | AQR Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0
Real estate term owner occupied | AQR Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0
Real estate term non-owner occupied
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	313,472	312,960
Total loans	311,869	311,302
Real estate term non-owner occupied | AQR Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	296,153	295,482
Real estate term non-owner occupied | AQR Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	17,319	17,478
Real estate term non-owner occupied | AQR Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0	0
Real estate term non-owner occupied | AQR Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0
Real estate term non-owner occupied | AQR Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0
Real estate term other
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	42,281	42,506
Total loans	41,980	42,200
Real estate term other | AQR Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	39,915	38,771
Real estate term other | AQR Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	1,190	2,559
Real estate term other | AQR Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	1,176	1,176
Real estate term other | AQR Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0
Real estate term other | AQR Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0
Consumer secured by 1st deeds of trust
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	15,574	16,198
Total loans	15,562	16,191
Consumer secured by 1st deeds of trust | AQR Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	15,240	15,860
Consumer secured by 1st deeds of trust | AQR Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	178	179
Consumer secured by 1st deeds of trust | AQR Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	87	159
Consumer secured by 1st deeds of trust | AQR Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0
Consumer secured by 1st deeds of trust | AQR Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	69
Consumer other
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	24,349	24,585
Total loans	24,526	24,761
Consumer other | AQR Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	24,232	24,464
Consumer other | AQR Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0	0
Consumer other | AQR Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	117	$ 121
Consumer other | AQR Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	0
Consumer other | AQR Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Total financing receivable	$ 0