Portfolio Loans Not Covered by Loss Share ("Non-covered") - Summary of Allowance for Loan Losses for Non-covered Loans by Portfolio Class and Category (Details) (USD $) In Thousands, unless otherwise specified	12 Months Ended
	Dec. 31, 2013	Dec. 31, 2012	Dec. 31, 2011
Non-Covered Loans
Allowance for Loan Losses [Roll Forward]
Balance	$ 34,330	$ 37,989	$ 42,759
Provision charged to expense	(642)	8,757	13,300
Losses charged off	(10,378)	(15,276)	(20,162)
Recoveries	3,979	2,860	2,092
Balance	27,289	34,330	37,989
Non-Covered Loans | Commercial & Industrial
Allowance for Loan Losses [Roll Forward]
Balance	10,064	11,945	12,727
Provision charged to expense	3,810	774	4,123
Losses charged off	(3,404)	(3,233)	(5,488)
Recoveries	1,776	578	583
Balance	12,246	10,064	11,945
Non-Covered Loans | Consumer & Other
Allowance for Loan Losses [Roll Forward]
Balance	31	14	93
Provision charged to expense	195	15	(136)
Losses charged off	(34)	0	(5)
Recoveries	0	2	62
Balance	192	31	14
Non-Covered Loans | Qualitative Adjustment
Allowance for Loan Losses [Roll Forward]
Balance	2,375	3,204	5,358
Provision charged to expense	(2,375)	(829)	(2,154)
Losses charged off	0	0	0
Recoveries	0	0	0
Balance	0	2,375	3,204
Non-Covered Loans | Commercial - Owner Occupied
Allowance for Loan Losses [Roll Forward]
Balance	4,192	6,297	5,060
Provision charged to expense	410	1,173	1,878
Losses charged off	(550)	(3,326)	(955)
Recoveries	44	48	314
Balance	4,096	4,192	6,297
Non-Covered Loans | Commercial - Investor Owned
Allowance for Loan Losses [Roll Forward]
Balance	10,403	6,751	5,629
Provision charged to expense	(94)	6,294	2,181
Losses charged off	(4,441)	(2,728)	(1,474)
Recoveries	732	86	415
Balance	6,600	10,403	6,751
Non-Covered Loans | Construction and Land Development
Allowance for Loan Losses [Roll Forward]
Balance	5,239	5,847	8,407
Provision charged to expense	(2,695)	3,081	7,652
Losses charged off	(896)	(4,384)	(10,627)
Recoveries	488	695	415
Balance	2,136	5,239	5,847
Non-Covered Loans | Residential
Allowance for Loan Losses [Roll Forward]
Balance	2,026	3,931	5,485
Provision charged to expense	107	(1,751)	(244)
Losses charged off	(1,053)	(1,605)	(1,613)
Recoveries	939	1,451	303
Balance	2,019	2,026	3,931
Unadvanced Commitment on Impaired Loan [Member]
Summary of Activity in the Allowance for Loan Losses by Portfolio Class and Category Based on Impairment Method [Line Items]
Estimated losses attributable to unadvanced commitments on impaired loans	$ 150	$ 124