Loans and Credit Quality - Nonaccrual Loans By Segment (Details) - USD ($) $ in Thousands	Sep. 30, 2016	Dec. 31, 2015
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	$ 10,885	$ 3,686
Government guarantees on nonaccrual loans	(1,624)	(1,561)
Net nonaccrual loans	9,261	2,125
Commercial
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	4,384	3,013
Real estate construction one-to-four family
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	3,972	0
Real estate construction other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	1,912	0
Real estate term owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	32	38
Real estate term non-owner occupied
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	209	359
Consumer secured by 1st deeds of trust
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	376	256
Consumer other
Nonaccrual Loans By Major Loan Type [Line Items]
Total nonaccrual loans	$ 0	$ 20