Exhibit 23.1

CONSENT OF INDEPENDENT REGISTERED PUBLIC ACCOUNTING FIRM

We consent to the incorporation by reference in Registration Statements No. 333-188137 on Form S-3 and Nos. 333-105946, 333-117973, 333-139736, 333-145030, and 333-195350 on Forms S-8 of First Security Group, Inc. of our report dated March 12, 2015 relating to the consolidated financial statements and effectiveness of internal control over financial reporting appearing in this Annual Report on Form 10-K.

/s/ Crowe Horwath LLP

Franklin, Tennessee

March 12, 2015