Loans and Credit Quality - Nonaccrual Loans by Segment (Details) - USD ($) $ in Thousands	Jun. 30, 2019	Dec. 31, 2018
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	$ 18,080	$ 15,210
Government guarantees on nonaccrual loans	(1,139)	(516)
Net nonaccrual loans	16,941	14,694
Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	11,207	12,671
Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	1,492	0
Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	3,844	1,694
Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	1,198	577
Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	210	220
Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	129	48
30-59 Days Past Due
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	2,093	1,329
Government guarantees on nonaccrual loans	0	(269)
Net nonaccrual loans	2,093	1,060
30-59 Days Past Due | Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	385	1,329
30-59 Days Past Due | Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0
30-59 Days Past Due | Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	1,087	0
30-59 Days Past Due | Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	621	0
30-59 Days Past Due | Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
30-59 Days Past Due | Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
60-89 Days Past Due
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	324
Government guarantees on nonaccrual loans	0	0
Net nonaccrual loans	0	324
60-89 Days Past Due | Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	324
60-89 Days Past Due | Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0
60-89 Days Past Due | Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
60-89 Days Past Due | Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
60-89 Days Past Due | Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
60-89 Days Past Due | Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
Greater Than 90 Days Past Due
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	6,983	3,597
Government guarantees on nonaccrual loans	(102)	0
Net nonaccrual loans	6,881	3,597
Greater Than 90 Days Past Due | Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	3,745	1,287
Greater Than 90 Days Past Due | Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0
Greater Than 90 Days Past Due | Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	2,568	1,694
Greater Than 90 Days Past Due | Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	577	577
Greater Than 90 Days Past Due | Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
Greater Than 90 Days Past Due | Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	93	39
Current
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	9,004	9,960
Government guarantees on nonaccrual loans	(1,037)	(247)
Net nonaccrual loans	7,967	9,713
Current | Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	7,077	9,731
Current | Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	1,492
Current | Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	189	0
Current | Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
Current | Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	210	220
Current | Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	$ 36	$ 9