Consolidated Statements of Operations (USD $) In Thousands, except Per Share data, unless otherwise specified	12 Months Ended
	Dec. 31, 2013	Dec. 31, 2012	Dec. 31, 2011
Interest Income:
Loans, including fees	$ 75,209	$ 93,255	$ 111,802
Securities:
Taxable	9,436	11,898	11,829
Exempt from Federal income taxes	2,915	3,606	4,270
Federal funds sold	38	39	95
Deposits in banks	1,985	1,254	621
Total interest income	89,583	110,052	128,617
Interest Expense:
Deposits	4,895	5,897	8,730
Securities sold under repurchase agreements and other short-term borrowings	27	98	246
Other borrowings	724	728	1,782
Total interest expense	5,646	6,723	10,758
Net Interest Income	83,937	103,329	117,859
Provision for loan losses	2,476	8,507	64,154
Net interest income after provision for loan losses	81,461	94,822	53,705
Non-Interest Income:
Service charges	5,813	6,279	7,013
Investment advisory fees	7,731	9,458	10,270
Other-than-temporary impairment loss:
Total impairment loss	(1,240)	(528)	(1,256)
Loss recognized in other comprehensive income			888
Net impairment loss recognized in earnings	(1,240)	(528)	(368)
Realized gains on securities available for sale, net			21
Gains (losses) on sales and revaluation of loans held for sale and other real estate owned, net	17	15,920	(427)
Other income	2,823	2,713	2,391
Total non-interest income	15,144	33,842	18,900
Non-Interest Expense:
Salaries and employee benefits	45,109	44,813	42,194
Occupancy	8,590	8,693	9,046
Professional services	6,846	7,587	7,399
Equipment	4,139	4,522	4,336
Business development	2,165	2,417	2,080
FDIC assessment	3,879	3,154	2,756
Goodwill impairment	18,700
Other operating expenses	10,673	11,352	12,344
Total non-interest expense	100,101	82,538	80,155
(Loss) income before income taxes	(3,496)	46,126	(7,550)
Income tax (benefit) expense	(4,626)	16,945	(5,413)
Net Income (Loss)	$ 1,130	$ 29,181	$ (2,137)
Basic earnings (loss) per common share	$ 0.06	$ 1.49	$ (0.11)
Diluted earnings (loss) per common share	$ 0.06	$ 1.49	$ (0.11)