Allowance for Loan Losses - Allowance for Loan Losses (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2020	Mar. 31, 2019
Accounts Receivable, Allowance for Credit Loss [Roll Forward]
Balance, beginning of period	$ 19,088	$ 19,519
Charge-Offs	(165)	(109)
Recoveries	34	49
Provision (benefit)	2,060	750
Balance, end of period	21,017	20,209
Balance, end of period: Individually evaluated for impairment	682	305
Balance, end of period: collectively evaluated for impairment	20,335	19,904
Commercial
Accounts Receivable, Allowance for Credit Loss [Roll Forward]
Balance, beginning of period	6,604	5,660
Charge-Offs	(151)	(109)
Recoveries	26	44
Provision (benefit)	1,790	883
Balance, end of period	8,269	6,478
Balance, end of period: Individually evaluated for impairment	682	212
Balance, end of period: collectively evaluated for impairment	7,587	6,266
Real estate construction one-to-four family
Accounts Receivable, Allowance for Credit Loss [Roll Forward]
Balance, beginning of period	643	675
Charge-Offs	0	0
Recoveries	0	0
Provision (benefit)	0	(16)
Balance, end of period	643	659
Balance, end of period: Individually evaluated for impairment	0	0
Balance, end of period: collectively evaluated for impairment	643	659
Real estate construction other
Accounts Receivable, Allowance for Credit Loss [Roll Forward]
Balance, beginning of period	1,017	1,275
Charge-Offs	0	0
Recoveries	0	0
Provision (benefit)	262	92
Balance, end of period	1,279	1,367
Balance, end of period: Individually evaluated for impairment	0	0
Balance, end of period: collectively evaluated for impairment	1,279	1,367
Real estate term owner occupied
Accounts Receivable, Allowance for Credit Loss [Roll Forward]
Balance, beginning of period	2,188	2,027
Charge-Offs	0	0
Recoveries	0	0
Provision (benefit)	242	293
Balance, end of period	2,430	2,320
Balance, end of period: Individually evaluated for impairment	0	0
Balance, end of period: collectively evaluated for impairment	2,430	2,320
Real estate term non-owner occupied
Accounts Receivable, Allowance for Credit Loss [Roll Forward]
Balance, beginning of period	5,180	5,799
Charge-Offs	0	0
Recoveries	0	0
Provision (benefit)	311	323
Balance, end of period	5,491	6,122
Balance, end of period: Individually evaluated for impairment	0	0
Balance, end of period: collectively evaluated for impairment	5,491	6,122
Real estate term other
Accounts Receivable, Allowance for Credit Loss [Roll Forward]
Balance, beginning of period	671	716
Charge-Offs	0	0
Recoveries	1	2
Provision (benefit)	39	126
Balance, end of period	711	844
Balance, end of period: Individually evaluated for impairment	0	54
Balance, end of period: collectively evaluated for impairment	711	790
Consumer secured by 1st deeds of trust
Accounts Receivable, Allowance for Credit Loss [Roll Forward]
Balance, beginning of period	270	306
Charge-Offs	0	0
Recoveries	0	0
Provision (benefit)	4	90
Balance, end of period	274	396
Balance, end of period: Individually evaluated for impairment	0	39
Balance, end of period: collectively evaluated for impairment	274	357
Consumer other
Accounts Receivable, Allowance for Credit Loss [Roll Forward]
Balance, beginning of period	436	426
Charge-Offs	(14)	0
Recoveries	7	3
Provision (benefit)	24	72
Balance, end of period	453	501
Balance, end of period: Individually evaluated for impairment	0	0
Balance, end of period: collectively evaluated for impairment	453	501
Unallocated
Accounts Receivable, Allowance for Credit Loss [Roll Forward]
Balance, beginning of period	2,079	2,635
Charge-Offs	0	0
Recoveries	0	0
Provision (benefit)	(612)	(1,113)
Balance, end of period	1,467	1,522
Balance, end of period: Individually evaluated for impairment	0	0
Balance, end of period: collectively evaluated for impairment	$ 1,467	$ 1,522