Summary of Significant Accounting Policies - Stockholders' Equity (Details)			12 Months Ended
	May 29, 2015	Mar. 31, 2015 USD ($) $ / shares shares	Dec. 31, 2016 USD ($) $ / shares shares	Dec. 31, 2015 USD ($) shares	Dec. 31, 2014 $ / shares shares
Debt Instrument [Line Items]
Proceeds from issuance of private placement		$ 76,000,000
Proceeds from issuance of private placement, net		68,200,000
Registration costs		7,800,000
Interest paid			$ 2,650,000	$ 7,358,000
Stock redeemed during period, value				19,000,000
Payments for capital contribution to subsidiary bank		36,000,000
Preferred stock authorized (in shares) | shares			200,000
Stock split conversion ratio	0.0333
Series C Preferred Stock
Debt Instrument [Line Items]
Stock redeemed during period, value		32,700,000
Payments for repurchase of redeemable preferred stock		$ 19,000,000	$ 0	$ 19,000,000
Preferred stock authorized (in shares) | shares			0	0
Preferred stock issued (in shares) | shares			0	0
Preferred stock, shares outstanding (in shares) | shares			0	0
Dividends in arrears			$ 0	$ 0
Stock redeemed (in shares) | shares		32,668
Series D Preferred Stock
Debt Instrument [Line Items]
Preferred stock authorized (in shares) | shares					2,636
Preferred stock issued (in shares) | shares					2,636
Liquidation preference (in dollars per share) | $ / shares					$ 1,000
Preferred stock, dividend rate					12.50%
Subordinated debentures
Debt Instrument [Line Items]
Interest paid		$ 4,900,000
Secured debt
Debt Instrument [Line Items]
Notes payable, repurchased face amount		10,300,000
Repayments of lines of credit		$ 8,500,000
Private placement offering
Debt Instrument [Line Items]
Number of shares sold (in shares) | shares		6,300,000
Offering price (in dollars per share) | $ / shares		$ 12.00
Series B mandatory redeemable preferred stock
Debt Instrument [Line Items]
Preferred stock authorized (in shares) | shares			1,092
Preferred stock issued (in shares) | shares			209	268
Preferred stock, shares outstanding (in shares) | shares			209	268
Dividend rate (in dollars per share) | $ / shares			$ 60.00
Preferred stock, redemption price (in dollars per share) | $ / shares			$ 1,000
Dividends in arrears			$ 0	$ 0