Exhibit 23.1

Consent of Independent Registered Public Accounting Firm

We consent to the incorporation by reference in the Registration Statements (Form S-3 Nos. 333-118966, 333-112598 and 333-90742 and Form S-8 Nos. 333-112561, 333-112218, 333-136890, 33-52892, 333-164618, 333-163245, 333-170708, and 333-170764) of Destination XL Group, Inc. of our reports dated March 15, 2013, with respect to the consolidated financial statements of Destination XL Group, Inc. and the effectiveness of internal control over financial reporting of Destination XL Group, Inc., included in this Annual Report (Form 10-K) for the year ended February 2, 2013.

/s/ Ernst & Young LLP

Boston, Massachusetts

March 15, 2013