Parent Company Condensed Financial Information - Condensed Statements of Cash Flows (Details) (USD $) In Thousands, unless otherwise specified	12 Months Ended
	Dec. 31, 2013	Dec. 31, 2012	Dec. 31, 2011
Cash flows from operating activities:
Net income (loss)	$ 18,826	$ (8,674)	$ (14,395)
Adjustments to reconcile net loss to net cash provided by operating activities:
Other-than-temporary impairment of investments	39	50	116
Change in other assets	(3,916)	5,526	6,648
Change in other liabilities	907	2,742	(1,913)
NET CASH PROVIDED BY OPERATING ACTIVITIES	40,987	33,732	65,331
Cash flows from investing activities:
NET CASH PROVIDED BY (USED IN) INVESTING ACTIVITIES	(7,755)	75,667	92,316
Cash flows from financing activities:
Issuance of preferred stock	0	40,592	0
Issuance of common stock	0	0	6,402
Dividends paid	(1,414)	(4,431)	(616)
Payment of fractional shares for the 1-for-3 stock split	(39)	0	0
Repurchase and cancellation of warrants	(20)	0	0
Net cash provided by (used in) financing activities	(130,633)	(63,921)	(294,509)
Net increase (decrease) in cash	(97,401)	45,478	(136,862)
Beginning of year	138,396		92,918
End of year	40,995	138,396
Parent Company
Cash flows from operating activities:
Net income (loss)	18,826	(8,674)	(14,395)
Adjustments to reconcile net loss to net cash provided by operating activities:
Equity (loss) in undistributed earnings of subsidiaries	(19,840)	7,684	13,511
Other-than-temporary impairment of investments	39	50	116
Loss on sale of other investments	0	0	80
Change in other assets	243	(18)	315
Change in other liabilities	(851)	113	270
NET CASH PROVIDED BY OPERATING ACTIVITIES	(1,583)	(845)	(103)
Cash flows from investing activities:
Purchase of investments	(267)	0	(250)
Maturities, call and repayments of investments	0	99	55
Proceeds from sale of other investments	0	0	460
Additional investment in bank subsidiary	0	(37,000)	0
NET CASH PROVIDED BY (USED IN) INVESTING ACTIVITIES	(267)	(36,901)	265
Cash flows from financing activities:
Issuance of preferred stock	0	40,592	0
Issuance of common stock	0	0	6,402
Dividends paid	(1,414)	(4,431)	(616)
Payment of fractional shares for the 1-for-3 stock split	(39)	0	0
Repurchase and cancellation of warrants	(20)	0	0
Net cash provided by (used in) financing activities	(1,473)	36,161	5,786
Net increase (decrease) in cash	(3,323)	(1,585)	5,948
Beginning of year	5,212		6,797
End of year	$ 1,889	$ 5,212