Operating Group Information - Sales by operating group (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Oct. 31, 2020	Nov. 02, 2019	Oct. 31, 2020	Nov. 02, 2019
Operating groups
Net sales	$ 175,135	$ 241,221	$ 527,466	$ 825,194
Operating Segments | Tommy Bahama
Operating groups
Net sales	94,905	127,023	277,143	480,623
Operating Segments | Lilly Pulitzer
Operating groups
Net sales	53,714	71,659	176,723	219,809
Operating Segments | Lanier Apparel
Operating groups
Net sales	10,810	28,758	29,985	75,378
Operating Segments | Southern Tide
Operating groups
Net sales	10,023	9,102	27,136	35,704
Corporate and Other
Operating groups
Net sales	5,683	4,679	16,479	13,680
Retail
Operating groups
Net sales	$ 47,415	$ 83,636	$ 134,701	$ 324,892
Net sales (as a percent)	27.00%	35.00%	26.00%	39.00%
Retail | Operating Segments | Tommy Bahama
Operating groups
Net sales (as a percent)	37.00%	46.00%	36.00%	48.00%
Retail | Operating Segments | Lilly Pulitzer
Operating groups
Net sales (as a percent)	22.00%	35.00%	19.00%	43.00%
Retail | Operating Segments | Southern Tide
Operating groups
Net sales (as a percent)	4.00%		3.00%
E-commerce
Operating groups
Net sales	$ 66,135	$ 62,310	$ 226,618	$ 180,736
Net sales (as a percent)	38.00%	26.00%	43.00%	22.00%
E-commerce | Operating Segments | Tommy Bahama
Operating groups
Net sales (as a percent)	27.00%	14.00%	34.00%	18.00%
E-commerce | Operating Segments | Lilly Pulitzer
Operating groups
Net sales (as a percent)	66.00%	55.00%	64.00%	36.00%
E-commerce | Operating Segments | Southern Tide
Operating groups
Net sales (as a percent)	24.00%	19.00%	29.00%	18.00%
E-commerce | Corporate and Other
Operating groups
Net sales (as a percent)	54.00%	55.00%	65.00%	59.00%
Restaurant
Operating groups
Net sales	$ 12,214	$ 17,325	$ 32,505	$ 61,457
Net sales (as a percent)	7.00%	7.00%	6.00%	8.00%
Restaurant | Operating Segments | Tommy Bahama
Operating groups
Net sales (as a percent)	13.00%	14.00%	12.00%	13.00%
Wholesale
Operating groups
Net sales	$ 49,221	$ 77,595	$ 132,768	$ 256,794
Net sales (as a percent)	28.00%	32.00%	25.00%	31.00%
Wholesale | Operating Segments | Tommy Bahama
Operating groups
Net sales (as a percent)	23.00%	26.00%	18.00%	21.00%
Wholesale | Operating Segments | Lilly Pulitzer
Operating groups
Net sales (as a percent)	12.00%	10.00%	17.00%	21.00%
Wholesale | Operating Segments | Lanier Apparel
Operating groups
Net sales (as a percent)	100.00%	100.00%	100.00%	100.00%
Wholesale | Operating Segments | Southern Tide
Operating groups
Net sales (as a percent)	72.00%	81.00%	68.00%	82.00%
Wholesale | Corporate and Other
Operating groups
Net sales (as a percent)	44.00%	39.00%	31.00%	34.00%
Other
Operating groups
Net sales	$ 150	$ 355	$ 874	$ 1,315
Other | Corporate and Other
Operating groups
Net sales (as a percent)	2.00%	6.00%	4.00%	7.00%