Exhibit 23.1

CONSENT OF INDEPENDENT REGISTERED PUBLIC ACCOUNTING FIRM

We consent to the incorporation by reference in Registration Statements No. 333-105946, 333-117973, 333-139736, and 333-145030 on Form S-8 of First Security Group, Inc. of our report dated March 14, 2014 relating to the consolidated financial statements and effectiveness of internal control over financial reporting appearing in this Annual Report on Form 10-K.

/s/ Crowe Horwath LLP

Brentwood, Tennessee

March 14, 2014