Parent Company Only Financial Information - Summary of Statement of Cash Flows (Details) (USD $) In Thousands, unless otherwise specified	12 Months Ended
	Dec. 31, 2014	Dec. 31, 2013	Dec. 31, 2012
Cash flows from operating activities
Interest received	$ 78,564	$ 73,849	$ 78,119
Interest paid	(4,508)	(5,584)	(9,003)
Income taxes received (paid)	(239)	(157)	2
Net cash provided (used) by operating activities	21,946	49,852	(2,062)
Cash flows from investing activities
Net cash provided by (used in) investment activities	(295,111)	(89,155)	(28,421)
Cash flows from financing activities
Issuance of common stock, net of related expense	24,637	46,977	0
Repurchase of stock warrants, including related expense	0	0	(81)
Stock based employment plans	142	190	196
Redemption of preferred stock	0	(50,000)	0
Dividends paid on preferred shares	0	(2,819)	(2,500)
Net cash provided by financing activities	182,080	55,940	38,389
Net increase (decrease) in cash and cash equivalents	(91,085)	16,637	7,906
Cash and cash equivalents at beginning of year	191,624	174,987	167,081
Cash and cash equivalents at end of year	100,539	191,624	174,987
RECONCILIATION OF INCOME (LOSS) TO CASH PROVIDED BY (USED IN) OPERATING ACTIVITIES
Net income (loss)	5,696	51,989	(710)
Adjustments to reconcile net income (loss) to net cash used in operating activities:
Change in other assets	2,954	792	(835)
Net cash provided (used) by operating activities	21,946	49,852	(2,062)
Parent Company [Member]
Cash flows from operating activities
Interest received	43	5	7
Interest paid	(1,058)	(957)	(1,045)
Dividends received	24	23	22
Income taxes received (paid)	573	1,797	(32)
Other	(964)	(494)	(703)
Net cash provided (used) by operating activities	(1,382)	374	(1,751)
Cash flows from investing activities
Decrease (increase) in securities purchased under agreement to resell, maturing within 30 days, net	(37,044)	2,130	422
Net cash provided by (used in) investment activities	(37,044)	2,130	422
Cash flows from financing activities
Issuance of common stock, net of related expense	24,637	46,977	0
Subordinated debt increase	13,208	0	0
Repurchase of stock warrants, including related expense	0	0	(81)
Stock based employment plans	142	190	196
Redemption of preferred stock	0	(50,000)	0
Dividends paid on preferred shares	0	(2,819)	(2,500)
Net cash provided by financing activities	37,987	(5,652)	(2,385)
Net increase (decrease) in cash and cash equivalents	(439)	(3,148)	(3,714)
Cash and cash equivalents at beginning of year	919	4,067	7,781
Cash and cash equivalents at end of year	480	919	4,067
RECONCILIATION OF INCOME (LOSS) TO CASH PROVIDED BY (USED IN) OPERATING ACTIVITIES
Net income (loss)	5,696	51,989	(710)
Adjustments to reconcile net income (loss) to net cash used in operating activities:
Equity in undistributed income of subsidiaries	(7,002)	(51,088)	(893)
Change in other assets	(76)	(527)	(148)
Net cash provided (used) by operating activities	$ (1,382)	$ 374	$ (1,751)