Loans and Credit Quality - Nonaccrual Loans By Segment (Details) - USD ($) $ in Thousands	Dec. 31, 2020	Dec. 31, 2019
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	$ 11,120	$ 15,356
Government guarantees on nonaccrual loans	(1,521)	(1,405)
Net nonaccrual loans	9,599	13,951
Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	5,576	9,153
Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	702	1,349
Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	3,497	3,489
Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	1,176	1,176
Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	63	68
Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	106	121
30-59 Days Past Due
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	68	1,937
Government guarantees on nonaccrual loans	(35)	(268)
Net nonaccrual loans	33	1,669
30-59 Days Past Due | Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	48	270
30-59 Days Past Due | Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	0	0
30-59 Days Past Due | Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	0	1,641
30-59 Days Past Due | Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	0	0
30-59 Days Past Due | Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	0	0
30-59 Days Past Due | Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	20	26
60-89 Days Past Due
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	229	474
Government guarantees on nonaccrual loans	(258)	0
Net nonaccrual loans	(29)	474
60-89 Days Past Due | Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	229	385
60-89 Days Past Due | Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	0	0
60-89 Days Past Due | Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	0	0
60-89 Days Past Due | Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	0	0
60-89 Days Past Due | Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	0	0
60-89 Days Past Due | Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	0	89
Greater Than 90 Days Past Due
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	7,071	6,010
Government guarantees on nonaccrual loans	0	0
Net nonaccrual loans	7,071	6,010
Greater Than 90 Days Past Due | Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	3,673	2,862
Greater Than 90 Days Past Due | Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	702	1,349
Greater Than 90 Days Past Due | Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	1,520	623
Greater Than 90 Days Past Due | Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	1,176	1,176
Greater Than 90 Days Past Due | Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	0	0
Greater Than 90 Days Past Due | Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	0	0
Current
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	3,752	6,935
Government guarantees on nonaccrual loans	(1,228)	(1,137)
Net nonaccrual loans	2,524	5,798
Current | Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	1,626	5,636
Current | Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	0	0
Current | Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	1,977	1,225
Current | Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	0	0
Current | Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	63	68
Current | Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	$ 86	$ 6