Credit Quality of Loans and Allowance for Loan Losses - Classes of Loans by Risk Rating (Details) - USD ($) $ in Thousands	Sep. 30, 2017	Dec. 31, 2016	Sep. 30, 2016
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 1,235,969	$ 1,284,082	$ 1,272,800
Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	447,482	459,574	463,031
Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	473,046	481,155	464,853
Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 920,528	$ 940,729
Percentage of Total Loans	100.00%	100.00%
Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 90,088	$ 100,959	96,365
Percentage of Total Loans	100.00%	100.00%
Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 155,676	$ 157,872	155,653
Percentage of Total Loans	100.00%	100.00%
Installment loans to individuals
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 63,148	$ 82,660	88,537
Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	760	1,095	1,449
Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	5,769	767	$ 2,912
Consumer and other commercial, total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 69,677	$ 84,522
Percentage of Total Loans	100.00%	100.00%
Pass | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 345,629	$ 346,246
Pass | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	421,071	420,970
Pass | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 766,700	$ 767,216
Percentage of Total Loans	83.29%	81.56%
Pass | Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 85,295	$ 100,775
Percentage of Total Loans	94.68%	99.82%
Pass | Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 149,548	$ 153,403
Percentage of Total Loans	96.06%	97.17%
Special mention | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 15,321	$ 22,611
Special mention | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	8,654	23,085
Special mention | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 23,975	$ 45,696
Percentage of Total Loans	2.60%	4.86%
Special mention | Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 1,834	$ 0
Percentage of Total Loans	2.04%	0.00%
Special mention | Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 1,839	$ 1,181
Percentage of Total Loans	1.18%	0.75%
Substandard | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 86,523	$ 90,300
Substandard | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	43,321	37,100
Substandard | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 129,844	$ 127,400
Percentage of Total Loans	14.11%	13.54%
Substandard | Real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 2,959	$ 184
Percentage of Total Loans	3.28%	0.18%
Substandard | Real estate - residential
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 4,289	$ 3,288
Percentage of Total Loans	2.76%	2.08%
Doubtful | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 9	$ 417
Doubtful | Real estate – commercial
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Doubtful | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 9	$ 417
Percentage of Total Loans	0.00%	0.04%
Performing | Installment loans to individuals
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 62,791	$ 82,103
Performing | Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	727	1,095
Performing | Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	5,769	767
Performing | Consumer and other commercial, total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 69,287	$ 83,965
Percentage of Total Loans	99.44%	99.34%
Nonperforming | Installment loans to individuals
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 357	$ 557
Nonperforming | Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	33	0
Nonperforming | Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Nonperforming | Consumer and other commercial, total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 390	$ 557
Percentage of Total Loans	0.56%	0.66%