Exhibit 23.1

CONSENT OF INDEPENDENT AUDITORS

We consent to the incorporation by reference in the Registration Statements (Form S-3 Nos. 333-112598, 333-90742 and 33-53358 and Form S-8 Nos. 33-112561, 33-112218, and 33-52892) of Casual Male Retail Group, Inc. of our report dated March 24, 2004, with respect to the consolidated financial statements and schedule of Casual Male Retail Group, Inc. included in this Annual Report (Form 10-K) for the year ended January 31, 2004.

/s/ ERNST & YOUNG LLP

Boston, Massachusetts

April 9, 2004