Allowance for Loan Losses - Balance of the Allowance Segregated by Segment and Class (Details) - USD ($) $ in Thousands	Mar. 31, 2020	Dec. 31, 2019	Mar. 31, 2019	Dec. 31, 2018
Financing Receivable, Allowance for Credit Loss [Line Items]
Individually evaluated for impairment	$ 682		$ 305
Collectively evaluated for impairment	20,335		19,904
Allowance for credit losses	21,017	$ 19,088	20,209	$ 19,519
AQR Substandard
Financing Receivable, Allowance for Credit Loss [Line Items]
Individually evaluated for impairment	682	17
Collectively evaluated for impairment	10	14
AQR Pass
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	18,292	16,399
AQR Special Mention
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	520	579
AQR Doubtful
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	46
Unallocated
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	1,467	2,079
Commercial
Financing Receivable, Allowance for Credit Loss [Line Items]
Individually evaluated for impairment	682		212
Collectively evaluated for impairment	7,587		6,266
Allowance for credit losses	8,269	6,604	6,478	5,660
Commercial | AQR Substandard
Financing Receivable, Allowance for Credit Loss [Line Items]
Individually evaluated for impairment	682	17
Collectively evaluated for impairment	5	9
Commercial | AQR Pass
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	7,473	6,514
Commercial | AQR Special Mention
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	63	64
Commercial | AQR Doubtful
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	46
Commercial | Unallocated
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	0	0
Real estate construction one-to-four family
Financing Receivable, Allowance for Credit Loss [Line Items]
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	643		659
Allowance for credit losses	643	643	659	675
Real estate construction one-to-four family | AQR Substandard
Financing Receivable, Allowance for Credit Loss [Line Items]
Individually evaluated for impairment	0	0
Collectively evaluated for impairment	0	0
Real estate construction one-to-four family | AQR Pass
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	617	588
Real estate construction one-to-four family | AQR Special Mention
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	26	55
Real estate construction one-to-four family | AQR Doubtful
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	0
Real estate construction one-to-four family | Unallocated
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	0	0
Real estate construction other
Financing Receivable, Allowance for Credit Loss [Line Items]
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	1,279		1,367
Allowance for credit losses	1,279	1,017	1,367	1,275
Real estate construction other | AQR Substandard
Financing Receivable, Allowance for Credit Loss [Line Items]
Individually evaluated for impairment	0	0
Collectively evaluated for impairment	0	0
Real estate construction other | AQR Pass
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	1,279	1,017
Real estate construction other | AQR Special Mention
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	0	0
Real estate construction other | AQR Doubtful
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	0
Real estate construction other | Unallocated
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	0	0
Real estate term owner occupied
Financing Receivable, Allowance for Credit Loss [Line Items]
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	2,430		2,320
Allowance for credit losses	2,430	2,188	2,320	2,027
Real estate term owner occupied | AQR Substandard
Financing Receivable, Allowance for Credit Loss [Line Items]
Individually evaluated for impairment	0	0
Collectively evaluated for impairment	0	0
Real estate term owner occupied | AQR Pass
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	2,365	2,125
Real estate term owner occupied | AQR Special Mention
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	65	63
Real estate term owner occupied | AQR Doubtful
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	0
Real estate term owner occupied | Unallocated
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	0	0
Real estate term non-owner occupied
Financing Receivable, Allowance for Credit Loss [Line Items]
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	5,491		6,122
Allowance for credit losses	5,491	5,180	6,122	5,799
Real estate term non-owner occupied | AQR Substandard
Financing Receivable, Allowance for Credit Loss [Line Items]
Individually evaluated for impairment	0	0
Collectively evaluated for impairment	0	0
Real estate term non-owner occupied | AQR Pass
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	5,151	4,829
Real estate term non-owner occupied | AQR Special Mention
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	340	351
Real estate term non-owner occupied | AQR Doubtful
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	0
Real estate term non-owner occupied | Unallocated
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	0	0
Real estate term other
Financing Receivable, Allowance for Credit Loss [Line Items]
Individually evaluated for impairment	0		54
Collectively evaluated for impairment	711		790
Allowance for credit losses	711	671	844	716
Real estate term other | AQR Substandard
Financing Receivable, Allowance for Credit Loss [Line Items]
Individually evaluated for impairment	0	0
Collectively evaluated for impairment	0	0
Real estate term other | AQR Pass
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	690	629
Real estate term other | AQR Special Mention
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	21	42
Real estate term other | AQR Doubtful
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	0
Real estate term other | Unallocated
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	0	0
Consumer secured by 1st deeds of trust
Financing Receivable, Allowance for Credit Loss [Line Items]
Individually evaluated for impairment	0		39
Collectively evaluated for impairment	274		357
Allowance for credit losses	274	270	396	306
Consumer secured by 1st deeds of trust | AQR Substandard
Financing Receivable, Allowance for Credit Loss [Line Items]
Individually evaluated for impairment	0	0
Collectively evaluated for impairment	0	0
Consumer secured by 1st deeds of trust | AQR Pass
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	269	266
Consumer secured by 1st deeds of trust | AQR Special Mention
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	5	4
Consumer secured by 1st deeds of trust | AQR Doubtful
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	0
Consumer secured by 1st deeds of trust | Unallocated
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	0	0
Consumer other
Financing Receivable, Allowance for Credit Loss [Line Items]
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	453		501
Allowance for credit losses	453	436	501	426
Consumer other | AQR Substandard
Financing Receivable, Allowance for Credit Loss [Line Items]
Individually evaluated for impairment	0	0
Collectively evaluated for impairment	5	5
Consumer other | AQR Pass
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	448	431
Consumer other | AQR Special Mention
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	0	0
Consumer other | AQR Doubtful
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	0
Consumer other | Unallocated
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	0	0
Unallocated
Financing Receivable, Allowance for Credit Loss [Line Items]
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	1,467		1,522
Allowance for credit losses	1,467	2,079	$ 1,522	$ 2,635
Unallocated | AQR Substandard
Financing Receivable, Allowance for Credit Loss [Line Items]
Individually evaluated for impairment	0	0
Collectively evaluated for impairment	0	0
Unallocated | AQR Pass
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	0	0
Unallocated | AQR Special Mention
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	0	0
Unallocated | AQR Doubtful
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	0
Unallocated | Unallocated
Financing Receivable, Allowance for Credit Loss [Line Items]
Collectively evaluated for impairment	$ 1,467	$ 2,079