Reportable Segment Information (Details) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 27, 2020 USD ($)	Jun. 29, 2019 USD ($)	Jun. 27, 2020 USD ($) segment	Jun. 29, 2019 USD ($)	Dec. 28, 2019 USD ($)
Segment Reporting Information [Line Items]
Number of reportable segments | segment			2
Net sales	$ 417,456	$ 526,026	$ 886,161	$ 1,005,482
Operating income (loss)	14,842	23,248	(8,886)	26,925
Interest expense, net	1,943	2,480	3,754	4,591
Income (loss) before income taxes	12,899	20,768	(12,640)	22,334
Depreciation and amortization	19,119	19,410	38,605	38,450
Capital expenditures (including capitalized software)	7,617	18,511	20,776	37,607
Identifiable assets	1,315,363		1,315,363		$ 1,452,512
Workplace furnishings
Segment Reporting Information [Line Items]
Net sales	308,081	409,512	646,467	763,023
Residential building products
Segment Reporting Information [Line Items]
Net sales	109,375	116,514	239,694	242,459
Operating segments | Workplace furnishings
Segment Reporting Information [Line Items]
Net sales	308,081	409,512	646,467	763,023
Operating income (loss)	7,785	18,749	(25,446)	17,018
Depreciation and amortization	10,782	11,247	22,113	22,307
Capital expenditures (including capitalized software)	4,293	12,347	11,394	22,666
Identifiable assets	741,876		741,876		874,913
Operating segments | Residential building products
Segment Reporting Information [Line Items]
Net sales	109,375	116,514	239,694	242,459
Operating income (loss)	14,365	13,362	35,036	30,970
Depreciation and amortization	2,318	2,174	4,624	4,230
Capital expenditures (including capitalized software)	206	2,577	3,179	7,575
Identifiable assets	383,642		383,642		364,653
General corporate
Segment Reporting Information [Line Items]
Operating income (loss)	(7,308)	(8,863)	(18,476)	(21,063)
Depreciation and amortization	6,019	5,989	11,868	11,913
Capital expenditures (including capitalized software)	3,118	$ 3,587	6,203	$ 7,366
Identifiable assets	$ 189,845		$ 189,845		$ 212,946