Allowance for Loan Losses - Allowance for Loan Losses (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2020	Jun. 30, 2019	Jun. 30, 2020	Jun. 30, 2019
Accounts Receivable, Allowance for Credit Loss [Roll Forward]
Balance, beginning of period	$ 21,017	$ 20,209	$ 19,088	$ 19,519
Charge-Offs	(804)	(68)	(969)	(177)
Recoveries	36	77	70	126
Provision (benefit)	404	300	2,464	1,050
Balance, end of period	20,653	20,518	20,653	20,518
Balance, end of period: Individually evaluated for impairment	31	6	31	6
Balance, end of period: collectively evaluated for impairment	20,622	20,512	20,622	20,512
Commercial
Accounts Receivable, Allowance for Credit Loss [Roll Forward]
Balance, beginning of period	8,269	6,478	6,604	5,660
Charge-Offs	(804)	(64)	(955)	(173)
Recoveries	30	48	56	92
Provision (benefit)	(129)	661	1,661	1,544
Balance, end of period	7,366	7,123	7,366	7,123
Balance, end of period: Individually evaluated for impairment	31	6	31	6
Balance, end of period: collectively evaluated for impairment	7,335	7,117	7,335	7,117
Real estate construction one-to-four family
Accounts Receivable, Allowance for Credit Loss [Roll Forward]
Balance, beginning of period	643	659	643	675
Charge-Offs	0	0	0	0
Recoveries	0	0	0	0
Provision (benefit)	47	80	47	64
Balance, end of period	690	739	690	739
Balance, end of period: Individually evaluated for impairment	0	0	0	0
Balance, end of period: collectively evaluated for impairment	690	739	690	739
Real estate construction other
Accounts Receivable, Allowance for Credit Loss [Roll Forward]
Balance, beginning of period	1,279	1,367	1,017	1,275
Charge-Offs	0	0	0	0
Recoveries	0	0	0	0
Provision (benefit)	(64)	(255)	198	(163)
Balance, end of period	1,215	1,112	1,215	1,112
Balance, end of period: Individually evaluated for impairment	0	0	0	0
Balance, end of period: collectively evaluated for impairment	1,215	1,112	1,215	1,112
Real estate term owner occupied
Accounts Receivable, Allowance for Credit Loss [Roll Forward]
Balance, beginning of period	2,430	2,320	2,188	2,027
Charge-Offs	0	0	0	0
Recoveries	0	0	0	0
Provision (benefit)	103	(39)	345	254
Balance, end of period	2,533	2,281	2,533	2,281
Balance, end of period: Individually evaluated for impairment	0	0	0	0
Balance, end of period: collectively evaluated for impairment	2,533	2,281	2,533	2,281
Real estate term non-owner occupied
Accounts Receivable, Allowance for Credit Loss [Roll Forward]
Balance, beginning of period	5,491	6,122	5,180	5,799
Charge-Offs	0	0	0	0
Recoveries	0	0	0	0
Provision (benefit)	(70)	109	241	432
Balance, end of period	5,421	6,231	5,421	6,231
Balance, end of period: Individually evaluated for impairment	0	0	0	0
Balance, end of period: collectively evaluated for impairment	5,421	6,231	5,421	6,231
Real estate term other
Accounts Receivable, Allowance for Credit Loss [Roll Forward]
Balance, beginning of period	711	844	671	716
Charge-Offs	0	0	0	0
Recoveries	1	25	1	27
Provision (benefit)	(10)	(108)	30	18
Balance, end of period	702	761	702	761
Balance, end of period: Individually evaluated for impairment	0	0	0	0
Balance, end of period: collectively evaluated for impairment	702	761	702	761
Consumer secured by 1st deed of trust
Accounts Receivable, Allowance for Credit Loss [Roll Forward]
Balance, beginning of period	274	396	270	306
Charge-Offs	0	0	0	0
Recoveries	0	0	0	0
Provision (benefit)	(16)	(74)	(12)	16
Balance, end of period	258	322	258	322
Balance, end of period: Individually evaluated for impairment	0	0	0	0
Balance, end of period: collectively evaluated for impairment	258	322	258	322
Consumer other
Accounts Receivable, Allowance for Credit Loss [Roll Forward]
Balance, beginning of period	453	501	436	426
Charge-Offs	0	(4)	(14)	(4)
Recoveries	5	4	13	7
Provision (benefit)	(11)	(15)	12	57
Balance, end of period	447	486	447	486
Balance, end of period: Individually evaluated for impairment	0	0	0	0
Balance, end of period: collectively evaluated for impairment	447	486	447	486
Unallocated
Accounts Receivable, Allowance for Credit Loss [Roll Forward]
Balance, beginning of period	1,467	1,522	2,079	2,635
Charge-Offs	0	0	0	0
Recoveries	0	0	0	0
Provision (benefit)	554	(59)	(58)	(1,172)
Balance, end of period	2,021	1,463	2,021	1,463
Balance, end of period: Individually evaluated for impairment	0	0	0	0
Balance, end of period: collectively evaluated for impairment	$ 2,021	$ 1,463	$ 2,021	$ 1,463