EXHIBIT 23

CONSENT OF INDEPENDENT REGISTERED PUBLIC ACCOUNTING FIRM

We consent to the incorporation by reference in the following Registration Statements:

• Registration Statement (Form S-8 No 333-134779) pertaining to The Dixie Group, Inc. 2006 Stock Awards Plan,

• Registration Statement (Form S-8 No 333-89994) pertaining to The Dixie Group, Inc. Stock Incentive Plan,

• Registration Statement (Form S-8 No 033-59564) pertaining to the Agreement and Plan of Merger by and among Dixie Yarns, Inc., Carriage Acquisitions, Inc. and Carriage Industries, Inc., dated as of November 3, 1992,

• Registration Statement (Form S-8 No 333-87534) pertaining to The Dixie Group, Inc. Stock Incentive Plan,

• Registration Statement (Form S-8 No 333-81163) pertaining to The Dixie Group, Inc. Incentive Stock Plan,

• Registration Statement (Form S-8 No 333-80971) pertaining to The Dixie Group, Inc. Core Leadership Team Stock Ownership Plan,

• Registration Statement (Form S-8 No 333-118504) pertaining to The Dixie Group, Inc. Directors Stock Plan,

• Registration Statement (Form S-8 No 333-168412) pertaining to The Dixie Group, Inc. Amended and Restated 2006 Stock Awards Plan,

• Registration Statement (Form S-8 No 333-188321) pertaining to The Dixie Group, Inc. Amended and Restated 2006 Stock Awards Plan,

• Registration Statement (Form S-3 No 333-194571) of The Dixie Group, Inc. pertaining to the offering 2,500,000 shares of common stock, and

• Registration Statement (Form S-8 No 333-211157) pertaining to The Dixie Group, Inc. 2016 Incentive Compensation Plan;

of our report dated March 13, 2018, with respect to the consolidated financial statements and schedule of The Dixie Group, Inc. included in this Annual Report (Form 10-K) of The Dixie Group, Inc. for the fiscal year ended December 30, 2017.

/s/ Dixon Hughes Goodman LLP

Atlanta, Georgia

March 13, 2018