Allowance for Loan Losses - Allowance for Loan Losses (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2019	Mar. 31, 2018
Allowance for Doubtful Accounts Receivable [Roll Forward]
Balance, beginning of period	$ 19,519	$ 21,461
Charge-Offs	(109)	(1,104)
Recoveries	49	92
Provision (benefit)	750	0
Balance, end of period	20,209	20,449
Balance, end of period: Individually evaluated for impairment	305	372
Balance, end of period: collectively evaluated for impairment	19,904	20,077
Commercial
Allowance for Doubtful Accounts Receivable [Roll Forward]
Balance, beginning of period	5,660	6,172
Charge-Offs	(109)	(965)
Recoveries	44	89
Provision (benefit)	883	388
Balance, end of period	6,478	5,684
Balance, end of period: Individually evaluated for impairment	212	370
Balance, end of period: collectively evaluated for impairment	6,266	5,314
Real estate construction one-to-four family
Allowance for Doubtful Accounts Receivable [Roll Forward]
Balance, beginning of period	675	629
Charge-Offs	0	0
Recoveries	0	0
Provision (benefit)	(16)	(21)
Balance, end of period	659	608
Balance, end of period: Individually evaluated for impairment	0	0
Balance, end of period: collectively evaluated for impairment	659	608
Real estate construction other
Allowance for Doubtful Accounts Receivable [Roll Forward]
Balance, beginning of period	1,275	1,566
Charge-Offs	0	0
Recoveries	0	0
Provision (benefit)	92	(497)
Balance, end of period	1,367	1,069
Balance, end of period: Individually evaluated for impairment	0	0
Balance, end of period: collectively evaluated for impairment	1,367	1,069
Real estate term owner occupied
Allowance for Doubtful Accounts Receivable [Roll Forward]
Balance, beginning of period	2,027	2,194
Charge-Offs	0	0
Recoveries	0	0
Provision (benefit)	293	(20)
Balance, end of period	2,320	2,174
Balance, end of period: Individually evaluated for impairment	0	0
Balance, end of period: collectively evaluated for impairment	2,320	2,174
Real estate term non-owner occupied
Allowance for Doubtful Accounts Receivable [Roll Forward]
Balance, beginning of period	5,799	6,043
Charge-Offs	0	0
Recoveries	0	0
Provision (benefit)	323	336
Balance, end of period	6,122	6,379
Balance, end of period: Individually evaluated for impairment	0	0
Balance, end of period: collectively evaluated for impairment	6,122	6,379
Real estate term other
Allowance for Doubtful Accounts Receivable [Roll Forward]
Balance, beginning of period	716	725
Charge-Offs	0	0
Recoveries	2	1
Provision (benefit)	126	299
Balance, end of period	844	1,025
Balance, end of period: Individually evaluated for impairment	54	0
Balance, end of period: collectively evaluated for impairment	790	1,025
Consumer secured by 1st deeds of trust
Allowance for Doubtful Accounts Receivable [Roll Forward]
Balance, beginning of period	306	315
Charge-Offs	0	(89)
Recoveries	0	1
Provision (benefit)	90	78
Balance, end of period	396	305
Balance, end of period: Individually evaluated for impairment	39	2
Balance, end of period: collectively evaluated for impairment	357	303
Consumer other
Allowance for Doubtful Accounts Receivable [Roll Forward]
Balance, beginning of period	426	307
Charge-Offs	0	(50)
Recoveries	3	1
Provision (benefit)	72	57
Balance, end of period	501	315
Balance, end of period: Individually evaluated for impairment	0	0
Balance, end of period: collectively evaluated for impairment	501	315
Unallocated
Allowance for Doubtful Accounts Receivable [Roll Forward]
Balance, beginning of period	2,635	3,510
Charge-Offs	0	0
Recoveries	0	0
Provision (benefit)	(1,113)	(620)
Balance, end of period	1,522	2,890
Balance, end of period: Individually evaluated for impairment	0	0
Balance, end of period: collectively evaluated for impairment	$ 1,522	$ 2,890