Fair Value Measurements - Estimate of the Fair Value of Financial Instruments Not Recorded at Fair Value on Balance Sheet (Details) (USD $) In Thousands, unless otherwise specified	Sep. 30, 2014	Dec. 31, 2013
Financial Assets:
State tax credits held for sale	$ 15,131	$ 16,491
Level 1
Financial Assets:
Portfolio loans, net	0	0
State tax credits held for sale	0	0
Financial Liabilities:
Deposits	1,951,370	1,902,038
Subordinated debentures	0	0
Federal Home Loan Bank advances	0	0
Other borrowings	0	0
Level 2
Financial Assets:
Portfolio loans, net	0	0
State tax credits held for sale	0	0
Financial Liabilities:
Deposits	0	0
Subordinated debentures	33,997	39,358
Federal Home Loan Bank advances	123,153	54,137
Other borrowings	187,123	214,377
Level 3
Financial Assets:
Portfolio loans, net	2,360,077	2,232,134
State tax credits held for sale	35,906	35,668
Financial Liabilities:
Deposits	562,048	638,784
Subordinated debentures	0	0
Federal Home Loan Bank advances	0	0
Other borrowings	0	0
Estimated fair value
Financial Assets:
Portfolio loans, net	2,360,077	2,232,134
State tax credits held for sale	35,906	35,668
Financial Liabilities:
Deposits	2,513,418	2,540,822
Subordinated debentures	33,997	39,358
Federal Home Loan Bank advances	123,153	54,137
Other borrowings	$ 187,123	$ 214,377