Credit Quality of Loans and Allowance for Loan Losses - Classes of Loans by Risk Rating (Details) - USD ($) $ in Thousands	Sep. 30, 2016	Dec. 31, 2015	Sep. 30, 2015
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 1,272,800	$ 1,263,645	$ 1,301,452
Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	463,031	454,028	482,452
Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	464,853	471,141	473,319
Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 927,884	$ 925,169
Percentage of Total Loans	100.00%	100.00%
Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 96,365	$ 74,952	74,279
Percentage of Total Loans	100.00%	100.00%
Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 155,653	$ 149,064	151,667
Percentage of Total Loans	100.00%	100.00%
Installment loans to individuals
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 88,537	$ 111,009	113,199
Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	1,449	1,968	4,790
Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	2,912	1,483	$ 1,746
Consumer and other commercial, total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 92,898	$ 114,460
Percentage of Total Loans	100.00%	100.00%
Pass | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 358,670	$ 383,897
Pass | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	403,961	412,141
Pass | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 762,631	$ 796,038
Percentage of Total Loans	82.20%	86.04%
Pass | Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 96,178	$ 74,794
Percentage of Total Loans	99.81%	99.79%
Pass | Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 151,053	$ 144,704
Percentage of Total Loans	97.04%	97.08%
Special mention | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 28,270	$ 32,506
Special mention | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	25,173	28,217
Special mention | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 53,443	$ 60,723
Percentage of Total Loans	5.76%	6.55%
Special mention | Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 0	$ 34
Percentage of Total Loans	0.00%	0.04%
Special mention | Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 1,227	$ 1,225
Percentage of Total Loans	0.79%	0.82%
Substandard | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 75,923	$ 37,353
Substandard | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	35,719	30,783
Substandard | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 111,642	$ 68,136
Percentage of Total Loans	12.03%	7.36%
Substandard | Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 187	$ 124
Percentage of Total Loans	0.19%	0.17%
Substandard | Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 3,373	$ 3,135
Percentage of Total Loans	2.17%	2.10%
Doubtful | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 168	$ 272
Doubtful | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Doubtful | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 168	$ 272
Percentage of Total Loans	0.02%	0.03%
Performing | Installment loans to individuals
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 88,048	$ 110,536
Performing | Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	1,449	1,968
Performing | Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	2,912	1,483
Performing | Consumer and other commercial, total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 92,409	$ 113,987
Percentage of Total Loans	99.47%	99.59%
Nonperforming | Installment loans to individuals
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 489	$ 473
Nonperforming | Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Nonperforming | Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Nonperforming | Consumer and other commercial, total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 489	$ 473
Percentage of Total Loans	0.53%	0.41%