Operating Group Information - Sales by operating group (Details) - USD ($) $ in Thousands	12 Months Ended
	Jan. 29, 2022	Jan. 30, 2021	Feb. 01, 2020
Operating groups
Net sales	$ 1,142,079	$ 748,833	$ 1,122,790
Operating Segments | Tommy Bahama
Operating groups
Net sales	724,305	419,817	676,652
Operating Segments | Lilly Pulitzer
Operating groups
Net sales	298,995	231,078	284,700
Operating Segments | Southern Tide
Operating groups
Net sales	54,050	34,664	46,409
Operating Segments | Lanier Apparel
Operating groups
Net sales	24,858	38,796	95,200
Corporate and Other
Operating groups
Net sales	39,871	24,478	19,829
Retail
Operating groups
Net sales	$ 443,015	$ 202,071	$ 440,803
Net sales (as a percent)	39.00%	27.00%	39.00%
Retail | Operating Segments | Tommy Bahama
Operating groups
Net sales (as a percent)	47.00%	37.00%	48.00%
Retail | Operating Segments | Lilly Pulitzer
Operating groups
Net sales (as a percent)	34.00%	20.00%	41.00%
Retail | Operating Segments | Southern Tide
Operating groups
Net sales (as a percent)	7.00%	4.00%
E-commerce
Operating groups
Net sales	$ 369,300	$ 323,900	$ 262,283
Net sales (as a percent)	32.00%	43.00%	23.00%
E-commerce | Operating Segments | Tommy Bahama
Operating groups
Net sales (as a percent)	25.00%	36.00%	20.00%
E-commerce | Operating Segments | Lilly Pulitzer
Operating groups
Net sales (as a percent)	50.00%	64.00%	38.00%
E-commerce | Operating Segments | Southern Tide
Operating groups
Net sales (as a percent)	26.00%	32.00%	21.00%
E-commerce | Corporate and Other
Operating groups
Net sales (as a percent)	54.00%	63.00%	59.00%
Restaurant
Operating groups
Net sales	$ 96,244	$ 48,428	$ 83,836
Net sales (as a percent)	8.00%	6.00%	8.00%
Restaurant | Operating Segments | Tommy Bahama
Operating groups
Net sales (as a percent)	13.00%	11.00%	12.00%
Wholesale
Operating groups
Net sales	$ 231,536	$ 173,209	$ 333,986
Net sales (as a percent)	20.00%	23.00%	30.00%
Wholesale | Operating Segments | Tommy Bahama
Operating groups
Net sales (as a percent)	15.00%	16.00%	20.00%
Wholesale | Operating Segments | Lilly Pulitzer
Operating groups
Net sales (as a percent)	16.00%	16.00%	21.00%
Wholesale | Operating Segments | Southern Tide
Operating groups
Net sales (as a percent)	67.00%	64.00%	79.00%
Wholesale | Operating Segments | Lanier Apparel
Operating groups
Net sales (as a percent)	100.00%	100.00%	100.00%
Wholesale | Corporate and Other
Operating groups
Net sales (as a percent)	42.00%	33.00%	34.00%
Other
Operating groups
Net sales	$ 1,984	$ 1,225	$ 1,882
Other | Corporate and Other
Operating groups
Net sales (as a percent)	4.00%	4.00%	7.00%