Loans and Credit Quality - Nonaccrual Loans By Segment (Details) - USD ($) $ in Thousands	Dec. 31, 2019	Dec. 31, 2018
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	$ 15,356	$ 15,210
Government guarantees on nonaccrual loans	(1,405)	(516)
Net nonaccrual loans	13,951	14,694
Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	9,153	12,671
Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	1,349	0
Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	3,489	1,694
Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	1,176	577
Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	68	220
Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	121	48
30-59 Days Past Due
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	1,937	1,329
Government guarantees on nonaccrual loans	(268)	(269)
Net nonaccrual loans	1,669	1,060
30-59 Days Past Due | Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	270	1,329
30-59 Days Past Due | Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	0
30-59 Days Past Due | Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	1,641	0
30-59 Days Past Due | Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	0	0
30-59 Days Past Due | Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	0	0
30-59 Days Past Due | Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	26	0
60-89 Days Past Due
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	474	324
Government guarantees on nonaccrual loans	0	0
Net nonaccrual loans	474	324
60-89 Days Past Due | Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	385	324
60-89 Days Past Due | Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	0
60-89 Days Past Due | Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	0	0
60-89 Days Past Due | Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	0	0
60-89 Days Past Due | Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	0	0
60-89 Days Past Due | Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	89	0
Greater Than 90 Days Past Due
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	6,010	3,597
Government guarantees on nonaccrual loans	0	0
Net nonaccrual loans	6,010	3,597
Greater Than 90 Days Past Due | Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	2,862	1,287
Greater Than 90 Days Past Due | Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	1,349
Greater Than 90 Days Past Due | Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	623	1,694
Greater Than 90 Days Past Due | Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	1,176	577
Greater Than 90 Days Past Due | Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	0	0
Greater Than 90 Days Past Due | Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	0	39
Current
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	6,935	9,960
Government guarantees on nonaccrual loans	(1,137)	(247)
Net nonaccrual loans	5,798	9,713
Current | Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	5,636	9,731
Current | Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	0
Current | Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	1,225	0
Current | Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	0	0
Current | Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	68	220
Current | Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	$ 6	$ 9