Selected Quarterly Financial Data (Unaudited) (Details) (USD $) In Thousands, except Per Share data, unless otherwise specified	3 Months Ended									12 Months Ended			30 Months Ended	36 Months Ended
	Dec. 31, 2013	Dec. 31, 2012	Sep. 30, 2012	Jun. 30, 2012	Mar. 31, 2012	Dec. 31, 2011	Sep. 30, 2011	Jun. 30, 2011	Mar. 31, 2011	Dec. 31, 2013	Dec. 31, 2012	Dec. 31, 2011	Dec. 31, 2013	Dec. 31, 2013
Quarterly Financial Information Disclosure [Abstract]
Interest income	$ 18,903		$ 18,470	$ 18,651	$ 18,275	$ 18,793	$ 19,446	$ 19,744	$ 20,989	$ 74,299	$ 78,972	$ 88,546
Interest expense	2,154		2,334	2,530	3,223	4,048	4,257	4,586	5,097	10,241	17,988	26,041
NET INTEREST INCOME	16,749		16,136	16,121	15,052	14,745	15,189	15,158	15,892	64,058	60,984	62,505
Provision for loan losses	(3,017)		40	55	237	31,554	4,251	2,218	2,351	(2,685)	40,374	20,843
NET INTEREST INCOME AFTER PROVISION FOR LOAN LOSSES	19,766		16,096	16,066	14,815	(16,809)	10,938	12,940	13,541	66,743	20,610	41,662
Other income	1,182		5,381	6,184	5,657	986	4,665	4,406	3,509	18,404	13,566	18,180
Other expense	13,712		14,150	14,843	13,215	22,708	14,792	15,732	13,579	55,920	66,811	67,543
INCOME (LOSS) BEFORE INCOME TAXES	7,236	38,500	7,327	7,407	7,257	(38,531)	811	1,614	3,471	29,227	(32,635)	(7,701)	43,900	(11,109)
Income taxes	2,579		2,616	2,598	2,608	(14,632)	54	(9,383)	0	10,401	(23,961)	6,694
NET INCOME (LOSS)	4,657		4,711	4,809	4,649	(23,899)	757	10,997	3,471	18,826	(8,674)	(14,395)
Preferred stock dividend and accretion of preferred stock discount	421		421	590	445	1,419	838	833	821	1,877	3,911	2,987
NET INCOME (LOSS) TO COMMON SHAREHOLDERS	$ 4,236		$ 4,290	$ 4,219	$ 4,204	$ (25,318)	$ (81)	$ 10,164	$ 2,650	$ 16,949	$ (12,585)	$ (17,382)
Earnings per share, basic	$ 0.30		$ 0.30	$ 0.30	$ 0.30	$ (3.63)	$ 0.00	$ 1.56	$ 0.42	$ 1.19	$ (1.91)	$ (2.86)
Earnings per share, diluted	$ 0.30		$ 0.30	$ 0.30	$ 0.30	$ (3.63)	$ 0.00	$ 1.56	$ 0.42	$ 1.19	$ (1.91)	$ (2.86)