Fair Value Measurements - Estimate of the Fair Value of Financial Instruments Not Recorded at Fair Value on Balance Sheet (Details) (USD $) In Thousands, unless otherwise specified	Dec. 31, 2013	Dec. 31, 2012
Financial Assets:
State tax credits, held for sale, carried at fair value	$ 16,491	$ 23,020
Fair Value, Level 1
Financial Assets:
Portfolio loans, net	0	0
State tax credits, held for sale, carried at fair value	0	0
Financial Liabilities:
Deposits	1,902,038	2,079,141
Subordinated debentures	0	0
Federal Home Loan Bank advances	0	0
Other borrowings	0	0
Fair Value, Level 2
Financial Assets:
Portfolio loans, net	0	0
State tax credits, held for sale, carried at fair value	0	0
Financial Liabilities:
Deposits	0	0
Subordinated debentures	39,358	65,840
Federal Home Loan Bank advances	54,137	89,301
Other borrowings	214,377	245,224
Fair Value, Level 3
Financial Assets:
Portfolio loans, net	2,232,134	2,266,834
State tax credits, held for sale, carried at fair value	35,668	43,802
Financial Liabilities:
Deposits	638,784	589,972
Subordinated debentures	0	0
Federal Home Loan Bank advances	0	0
Other borrowings	0	0
Estimated fair value
Financial Assets:
Portfolio loans, net	2,232,134	2,266,834
State tax credits, held for sale, carried at fair value	35,668	43,802
Financial Liabilities:
Deposits	2,540,822	2,669,113
Subordinated debentures	39,358	65,840
Federal Home Loan Bank advances	54,137	89,301
Other borrowings	$ 214,377	$ 245,224