Loans and Credit Quality - Nonaccrual Loans by Segment (Details) - USD ($) $ in Thousands	Jun. 30, 2020	Dec. 31, 2019
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	$ 14,365	$ 15,356
Government guarantees on nonaccrual loans	(1,635)	(1,405)
Net nonaccrual loans	12,730	13,951
Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	8,362	9,153
Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	702	1,349
Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	3,947	3,489
Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	1,176	1,176
Consumer secured by 1st deed of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	66	68
Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	112	121
30-59 Days Past Due
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	354	1,937
Government guarantees on nonaccrual loans	(73)	(268)
Net nonaccrual loans	281	1,669
30-59 Days Past Due | Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	354	270
30-59 Days Past Due | Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
30-59 Days Past Due | Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	1,641
30-59 Days Past Due | Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
30-59 Days Past Due | Consumer secured by 1st deed of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
30-59 Days Past Due | Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	26
60-89 Days Past Due
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	60	474
Government guarantees on nonaccrual loans	0	0
Net nonaccrual loans	60	474
60-89 Days Past Due | Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	385
60-89 Days Past Due | Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
60-89 Days Past Due | Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	60	0
60-89 Days Past Due | Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
60-89 Days Past Due | Consumer secured by 1st deed of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
60-89 Days Past Due | Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	89
Greater Than 90 Days Past Due
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	10,154	6,010
Government guarantees on nonaccrual loans	0	0
Net nonaccrual loans	10,154	6,010
Greater Than 90 Days Past Due | Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	6,031	2,862
Greater Than 90 Days Past Due | Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	702	1,349
Greater Than 90 Days Past Due | Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	2,245	623
Greater Than 90 Days Past Due | Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	1,176	1,176
Greater Than 90 Days Past Due | Consumer secured by 1st deed of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
Greater Than 90 Days Past Due | Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
Current
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	3,797	6,935
Government guarantees on nonaccrual loans	(1,562)	(1,137)
Net nonaccrual loans	2,235	5,798
Current | Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	1,977	5,636
Current | Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
Current | Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	1,642	1,225
Current | Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
Current | Consumer secured by 1st deed of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	66	68
Current | Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	$ 112	$ 6