Accumulated Other Comprehensive Income - Reclassifications (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2016	Sep. 30, 2016	Jun. 30, 2016	Mar. 31, 2016	Dec. 31, 2015	Sep. 30, 2015	Jun. 30, 2015	Mar. 31, 2015	Dec. 31, 2016	Dec. 31, 2015	Dec. 31, 2014
Reclassifications from AOCI
Net gain (loss) on securities, other assets and OREO									$ 121	$ 369	$ 2,597
Interest income	$ 32,779	$ 32,308	$ 31,500	$ 31,195	$ 31,081	$ 30,177	$ 30,044	$ 29,964	127,782	121,266	114,317
Interest and fees on loans									114,612	107,622	98,627
Interest expense on subordinated debentures									(7,383)	(5,736)	(4,122)
(Income) loss before taxes	$ (11,825)	$ (13,812)	$ (11,729)	$ (9,715)	$ (5,114)	$ (9,291)	$ (10,596)	$ (10,745)	(47,081)	(35,746)	(43,816)
Provision for income taxes									(12,182)	(9,606)	(15,018)
Total reclassifications out of AOCI									34,899	25,749	28,433
Reclassification out of AOCI
Reclassifications from AOCI
Total reclassifications out of AOCI									537	706	(97)
Reclassification out of AOCI | Cash flow hedges | Unrealized (gain) loss on derivatives
Reclassifications from AOCI
Interest and fees on loans									630	927	905
Interest expense on subordinated debentures									(1,690)	(2,009)	(2,215)
(Income) loss before taxes									(1,060)	(1,082)	(1,310)
Provision for income taxes									404	411	498
Total reclassifications out of AOCI									(656)	(671)	(812)
Reclassification out of AOCI | Available for sale securities | Unrealized (gain) loss on securities
Reclassifications from AOCI
Net gain (loss) on securities, other assets and OREO									94	318	1,154
Provision for income taxes									36	121	439
Total reclassifications out of AOCI									58	197	$ 715
Reclassification out of AOCI | Held to maturity securities | Unrealized (gain) loss on securities
Reclassifications from AOCI
Interest income									1,831	1,904
Provision for income taxes									(696)	(724)
Total reclassifications out of AOCI									$ 1,135	$ 1,180