Credit Quality of Loans and Allowance for Loan Losses - Classes of Loans by Risk Rating (Details) - USD ($) $ in Thousands	Mar. 31, 2018	Dec. 31, 2017	Mar. 31, 2017
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 1,137,255	$ 1,183,426	$ 1,272,000
Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	401,048	435,207	469,815
Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	438,779	448,406	464,859
Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 839,827	$ 883,613
Percentage of Total Loans	100.00%	100.00%
Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 94,679	$ 90,287	100,248
Percentage of Total Loans	100.00%	100.00%
Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 145,671	$ 146,751	159,426
Percentage of Total Loans	100.00%	100.00%
Installment loans to individuals
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 50,888	$ 56,398	75,258
Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	692	732	969
Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	5,498	5,645	$ 1,425
Consumer and other commercial, total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 57,078	$ 62,775
Percentage of Total Loans	100.00%	100.00%
Pass | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 326,697	$ 358,373
Pass | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	393,327	411,280
Pass | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 720,024	$ 769,653
Percentage of Total Loans	85.73%	87.10%
Pass | Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 92,978	$ 89,323
Percentage of Total Loans	98.20%	98.93%
Pass | Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 140,883	$ 144,250
Percentage of Total Loans	96.71%	98.30%
Special mention | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 8,718	$ 9,687
Special mention | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	5,798	3,823
Special mention | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 14,516	$ 13,510
Percentage of Total Loans	1.73%	1.53%
Special mention | Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 167	$ 600
Percentage of Total Loans	0.18%	0.66%
Special mention | Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 854	$ 1,233
Percentage of Total Loans	0.59%	0.84%
Substandard | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 65,633	$ 67,147
Substandard | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	39,654	33,303
Substandard | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 105,287	$ 100,450
Percentage of Total Loans	12.54%	11.37%
Substandard | Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 1,534	$ 364
Percentage of Total Loans	1.62%	0.40%
Substandard | Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 3,934	$ 1,268
Percentage of Total Loans	2.70%	0.86%
Performing | Installment loans to individuals
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 50,837	$ 56,041
Performing | Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	692	699
Performing | Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	5,498	5,645
Performing | Consumer and other commercial, total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 57,027	$ 62,385
Percentage of Total Loans	99.91%	99.38%
Nonperforming | Installment loans to individuals
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 51	$ 357
Nonperforming | Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	33
Nonperforming | Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Nonperforming | Consumer and other commercial, total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 51	$ 390
Percentage of Total Loans	0.09%	0.62%