Credit Quality of Loans and Allowance for Loan Losses - Classes of Loans by Risk Rating (Details) - USD ($) $ in Thousands	Sep. 30, 2015	Dec. 31, 2014	Sep. 30, 2014
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 1,301,452	$ 1,284,431	$ 1,248,373
Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	482,452	467,147	452,065
Commercial real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	55,688	43,510
Commercial real estate - other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	473,319	467,172
Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 1,011,459	$ 977,829
Percentage of Total Loans	100.00%	100.00%
Residential - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 18,591	$ 25,067
Residential - prime
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	151,667	154,602
Residential, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 170,258	$ 179,669
Percentage of Total Loans	100.00%	100.00%
Consumer - credit card
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 5,759	$ 6,012
Consumer - other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	107,440	113,316
Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	4,790	4,857	5,285
Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	1,746	2,748	$ 3,523
Consumer, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 119,735	$ 126,933
Percentage of Total Loans	100.00%	100.00%
Pass | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 417,600	$ 456,221
Pass | Commercial real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	55,525	43,320
Pass | Commercial real estate - other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	417,095	440,281
Pass | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 890,220	$ 939,822
Percentage of Total Loans	88.02%	96.11%
Pass | Residential - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 18,419	$ 25,067
Pass | Residential - prime
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	147,721	150,664
Pass | Residential, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 166,140	$ 175,731
Percentage of Total Loans	97.58%	97.81%
Special mention | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 23,104	$ 4,861
Special mention | Commercial real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	34	132
Special mention | Commercial real estate - other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	20,277	7,120
Special mention | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 43,415	$ 12,113
Percentage of Total Loans	4.29%	1.24%
Special mention | Residential - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 0	$ 0
Special mention | Residential - prime
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	1,326	1,184
Special mention | Residential, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 1,326	$ 1,184
Percentage of Total Loans	0.78%	0.66%
Substandard | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 41,521	$ 5,541
Substandard | Commercial real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	129	58
Substandard | Commercial real estate - other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	35,947	19,771
Substandard | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 77,597	$ 25,370
Percentage of Total Loans	7.67%	2.60%
Substandard | Residential - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 172	$ 0
Substandard | Residential - prime
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	2,620	2,754
Substandard | Residential, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 2,792	$ 2,754
Percentage of Total Loans	1.64%	1.53%
Doubtful | Commercial, financial and agricultural
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 227	$ 524
Doubtful | Commercial real estate - construction
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Doubtful | Commercial real estate - other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Doubtful | Commercial, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 227	$ 524
Percentage of Total Loans	0.02%	0.05%
Performing | Consumer - credit card
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 5,721	$ 5,995
Performing | Consumer - other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	107,022	112,893
Performing | Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	4,790	4,857
Performing | Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	1,746	2,748
Performing | Consumer, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 119,279	$ 126,493
Percentage of Total Loans	99.62%	99.65%
Nonperforming | Consumer - credit card
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 38	$ 17
Nonperforming | Consumer - other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	418	423
Nonperforming | Lease financing receivable
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Nonperforming | Other
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	0	0
Nonperforming | Consumer, Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Loans	$ 456	$ 440
Percentage of Total Loans	0.38%	0.35%