Loans and Credit Quality - Nonaccrual Loans by Segment (Details) - USD ($) $ in Thousands	Mar. 31, 2019	Dec. 31, 2018
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	$ 19,516	$ 15,210
Government guarantees on nonaccrual loans	(1,038)	(516)
Net nonaccrual loans	18,478	14,694
Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	11,686	12,671
Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	2,423	0
Real estate construction other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	3,774	1,694
Real estate term non-owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	1,227	577
Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	360	220
Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	46	48
30-59 Days Past Due
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	654	1,329
Government guarantees on nonaccrual loans	0	(269)
Net nonaccrual loans	654	1,060
30-59 Days Past Due | Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	261	1,329
30-59 Days Past Due | Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0
30-59 Days Past Due | Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	393	0
30-59 Days Past Due | Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
30-59 Days Past Due | Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
30-59 Days Past Due | Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
60-89 Days Past Due
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	1,337	324
Government guarantees on nonaccrual loans	(269)	0
Net nonaccrual loans	1,068	324
60-89 Days Past Due | Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	1,337	324
60-89 Days Past Due | Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0
60-89 Days Past Due | Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
60-89 Days Past Due | Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
60-89 Days Past Due | Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
60-89 Days Past Due | Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
Greater Than 90 Days Past Due
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	5,593	3,597
Government guarantees on nonaccrual loans	0	0
Net nonaccrual loans	5,593	3,597
Greater Than 90 Days Past Due | Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	2,559	1,287
Greater Than 90 Days Past Due | Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0
Greater Than 90 Days Past Due | Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	2,418	1,694
Greater Than 90 Days Past Due | Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	577	577
Greater Than 90 Days Past Due | Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	0	0
Greater Than 90 Days Past Due | Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	39	39
Current
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	11,932	9,960
Government guarantees on nonaccrual loans	(769)	(247)
Net nonaccrual loans	11,163	9,713
Current | Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	7,529	9,731
Current | Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	2,423
Current | Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	963	0
Current | Real estate term other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	650	0
Current | Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	360	220
Current | Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonperforming loans	$ 7	$ 9