Loans and Credit Quality - Nonaccrual Loans by Segment (Details) - USD ($) $ in Thousands	Sep. 30, 2019	Dec. 31, 2018
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	$ 17,442	$ 15,210
Government guarantees on nonaccrual loans	(1,935)	(516)
Net nonaccrual loans	15,507	14,694
Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	10,868	12,671
Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	1,473	0
Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	3,776	1,694
Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	1,198	577
Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	220
Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	127	48
30-59 Days Past Due
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	145	1,329
Government guarantees on nonaccrual loans	(75)	(269)
Net nonaccrual loans	70	1,060
30-59 Days Past Due | Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	145	1,329
30-59 Days Past Due | Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0
30-59 Days Past Due | Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
30-59 Days Past Due | Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
30-59 Days Past Due | Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans		0
30-59 Days Past Due | Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
60-89 Days Past Due
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	1,642	324
Government guarantees on nonaccrual loans	0	0
Net nonaccrual loans	1,642	324
60-89 Days Past Due | Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	169	324
60-89 Days Past Due | Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	1,473
60-89 Days Past Due | Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
60-89 Days Past Due | Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
60-89 Days Past Due | Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans		0
60-89 Days Past Due | Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
Greater Than 90 Days Past Due
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	8,081	3,597
Government guarantees on nonaccrual loans	(102)	0
Net nonaccrual loans	7,979	3,597
Greater Than 90 Days Past Due | Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	3,857	1,287
Greater Than 90 Days Past Due | Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0
Greater Than 90 Days Past Due | Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	2,933	1,694
Greater Than 90 Days Past Due | Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	1,198	577
Greater Than 90 Days Past Due | Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans		0
Greater Than 90 Days Past Due | Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	93	39
Current
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	7,574	9,960
Government guarantees on nonaccrual loans	(1,758)	(247)
Net nonaccrual loans	5,816	9,713
Current | Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	6,697	9,731
Current | Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0
Current | Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	843	0
Current | Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
Current | Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans		220
Current | Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	$ 34	$ 9