Exhibit 23

Consent of Elliott Davis, LLC

CONSENT OF INDEPENDENT REGISTERED PUBLIC ACCOUNTING FIRM

The Board of Directors

Security Federal Corporation

We consent to incorporation by reference in the Registration Statement No. 333-31500, 333-102337 and 333-136813 on Form S-8 of our report dated June 22, 2012, relating to the consolidated balance sheet of Security Federal Corporation and subsidiaries as of March 31, 2012 and the related consolidated statements of income, changes in shareholders’ equity and comprehensive income and cash flows for the year then ended, which report appears in the March 31, 2012 Annual Report on Form 10-K.

/s/ Elliott Davis, LLC

Columbia, South Carolina

June 25, 2012