Loans and Credit Quality - Nonaccrual Loans by Segment (Details) - USD ($) $ in Thousands	Mar. 31, 2020	Dec. 31, 2019
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	$ 15,074	$ 15,356
Government guarantees on nonaccrual loans	(1,671)	(1,405)
Net nonaccrual loans	13,403	13,951
Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	9,340	9,153
Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	915	1,349
Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	3,459	3,489
Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	1,176	1,176
Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	67	68
Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	117	121
30-59 Days Past Due
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	5,344	1,937
Government guarantees on nonaccrual loans	(340)	(268)
Net nonaccrual loans	5,004	1,669
30-59 Days Past Due | Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	4,285	270
30-59 Days Past Due | Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
30-59 Days Past Due | Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	1,029	1,641
30-59 Days Past Due | Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
30-59 Days Past Due | Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
30-59 Days Past Due | Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	30	26
60-89 Days Past Due
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	945	474
Government guarantees on nonaccrual loans	0	0
Net nonaccrual loans	945	474
60-89 Days Past Due | Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	308	385
60-89 Days Past Due | Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
60-89 Days Past Due | Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	637	0
60-89 Days Past Due | Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
60-89 Days Past Due | Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
60-89 Days Past Due | Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	89
Greater Than 90 Days Past Due
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	6,735	6,010
Government guarantees on nonaccrual loans	0	0
Net nonaccrual loans	6,735	6,010
Greater Than 90 Days Past Due | Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	3,030	2,862
Greater Than 90 Days Past Due | Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	915	1,349
Greater Than 90 Days Past Due | Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	1,614	623
Greater Than 90 Days Past Due | Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	1,176	1,176
Greater Than 90 Days Past Due | Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
Greater Than 90 Days Past Due | Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
Current
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	2,050	6,935
Government guarantees on nonaccrual loans	(1,331)	(1,137)
Net nonaccrual loans	719	5,798
Current | Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	1,717	5,636
Current | Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
Current | Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	179	1,225
Current | Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
Current | Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	67	68
Current | Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	$ 87	$ 6