Operating Group Information - Sales by operating group (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Feb. 01, 2020	Nov. 02, 2019	Aug. 03, 2019	May 04, 2019	Feb. 02, 2019	Nov. 03, 2018	Aug. 04, 2018	May 05, 2018	Feb. 01, 2020	Feb. 02, 2019	Feb. 03, 2018
Operating groups
Net sales	$ 297,596	$ 241,221	$ 302,000	$ 281,973	$ 298,535	$ 233,662	$ 302,641	$ 272,628	$ 1,122,790	$ 1,107,466	$ 1,086,211
Operating Segments | Tommy Bahama
Operating groups
Net sales									676,652	675,358	686,021
Operating Segments | Lilly Pulitzer
Operating groups
Net sales									284,700	272,299	248,931
Operating Segments | Lanier Apparel
Operating groups
Net sales									97,251	100,471	106,852
Operating Segments | Southern Tide
Operating groups
Net sales									46,409	45,248	40,940
Corporate and Other
Operating groups
Net sales									17,778	14,090	3,467
Retail
Operating groups
Net sales									$ 440,803	$ 439,556	$ 427,439
Net sales (as a percent)									39.00%	40.00%	39.00%
Retail | Operating Segments | Tommy Bahama
Operating groups
Net sales (as a percent)									48.00%	48.00%	49.00%
Retail | Operating Segments | Lilly Pulitzer
Operating groups
Net sales (as a percent)									41.00%	42.00%	38.00%
E-commerce
Operating groups
Net sales									$ 262,283	$ 239,034	$ 205,475
Net sales (as a percent)									23.00%	21.00%	19.00%
E-commerce | Operating Segments | Tommy Bahama
Operating groups
Net sales (as a percent)									20.00%	18.00%	16.00%
E-commerce | Operating Segments | Lilly Pulitzer
Operating groups
Net sales (as a percent)									38.00%	36.00%	34.00%
E-commerce | Operating Segments | Lanier Apparel
Operating groups
Net sales (as a percent)									1.00%
E-commerce | Operating Segments | Southern Tide
Operating groups
Net sales (as a percent)									21.00%	18.00%	19.00%
E-commerce | Corporate and Other
Operating groups
Net sales (as a percent)									60.00%	54.00%	23.00%
Restaurant
Operating groups
Net sales									$ 83,836	$ 84,530	$ 83,900
Net sales (as a percent)									8.00%	8.00%	8.00%
Restaurant | Operating Segments | Tommy Bahama
Operating groups
Net sales (as a percent)									12.00%	13.00%	12.00%
Wholesale
Operating groups
Net sales									$ 333,986	$ 341,615	$ 366,123
Net sales (as a percent)									30.00%	31.00%	34.00%
Wholesale | Operating Segments | Tommy Bahama
Operating groups
Net sales (as a percent)									20.00%	21.00%	23.00%
Wholesale | Operating Segments | Lilly Pulitzer
Operating groups
Net sales (as a percent)									21.00%	22.00%	28.00%
Wholesale | Operating Segments | Lanier Apparel
Operating groups
Net sales (as a percent)									99.00%	100.00%	100.00%
Wholesale | Operating Segments | Southern Tide
Operating groups
Net sales (as a percent)									79.00%	82.00%	81.00%
Wholesale | Corporate and Other
Operating groups
Net sales (as a percent)									32.00%	30.00%	16.00%
Other
Operating groups
Net sales									$ 1,882	$ 2,731	$ 3,274
Other | Corporate and Other
Operating groups
Net sales (as a percent)									8.00%	16.00%	61.00%