Exhibit 23.1

Consent of Independent Registered Public Accounting Firm

We consent to the reference to our firm under the caption “Experts” in Amendment No. 1 to the Registration Statement (Form S-3 No. 214783) and related Prospectus of Delta Apparel, Inc. and subsidiaries for the registration of common stock, preferred stock, debt securities, warrants, and units and to the incorporation by reference therein of our reports dated November 29, 2016, with respect to the consolidated financial statements and schedule of Delta Apparel, Inc. and subsidiaries and the effectiveness of internal control over financial reporting of Delta Apparel, Inc. and subsidiaries included in its Annual Report (Form 10-K) for the year ended October 1, 2016, filed with the Securities and Exchange Commission.

/s/ Ernst & Young LLP

Atlanta, Georgia

December 19, 2016