Operating Group Information (Details 2) - USD ($) $ in Thousands	3 Months Ended
	May 04, 2019	May 05, 2018
Operating group information
Net sales	$ 281,973	$ 272,628
Operating Groups | Tommy Bahama
Operating group information
Net sales	164,730	167,132
Operating Groups | Lilly Pulitzer
Operating group information
Net sales	72,595	68,627
Operating Groups | Lanier Apparel
Operating group information
Net sales	26,589	19,909
Operating Groups | Southern Tide
Operating group information
Net sales	14,134	13,472
Corporate and Other
Operating group information
Net sales	3,925	3,488
Retail
Operating group information
Net sales	$ 108,006	$ 107,735
Revenue, net, percentage	38.00%	40.00%
Retail | Operating Groups | Tommy Bahama
Operating group information
Revenue, net, percentage	47.00%	46.00%
Retail | Operating Groups | Lilly Pulitzer
Operating group information
Revenue, net, percentage	42.00%	44.00%
Retail | Operating Groups | Lanier Apparel
Operating group information
Revenue, net, percentage	0.00%	0.00%
Retail | Operating Groups | Southern Tide
Operating group information
Revenue, net, percentage	0.00%	0.00%
Retail | Corporate and Other
Operating group information
Revenue, net, percentage	0.00%	0.00%
E-commerce
Operating group information
Net sales	$ 47,989	$ 44,522
Revenue, net, percentage	17.00%	16.00%
E-commerce | Operating Groups | Tommy Bahama
Operating group information
Revenue, net, percentage	15.00%	15.00%
E-commerce | Operating Groups | Lilly Pulitzer
Operating group information
Revenue, net, percentage	25.00%	24.00%
E-commerce | Operating Groups | Lanier Apparel
Operating group information
Revenue, net, percentage	1.00%	0.00%
E-commerce | Operating Groups | Southern Tide
Operating group information
Revenue, net, percentage	13.00%	12.00%
E-commerce | Corporate and Other
Operating group information
Revenue, net, percentage	61.00%	60.00%
Restaurant
Operating group information
Net sales	$ 23,601	$ 25,293
Revenue, net, percentage	9.00%	9.00%
Restaurant | Operating Groups | Tommy Bahama
Operating group information
Revenue, net, percentage	14.00%	15.00%
Restaurant | Operating Groups | Lilly Pulitzer
Operating group information
Revenue, net, percentage	0.00%	0.00%
Restaurant | Operating Groups | Lanier Apparel
Operating group information
Revenue, net, percentage	0.00%	0.00%
Restaurant | Operating Groups | Southern Tide
Operating group information
Revenue, net, percentage	0.00%	0.00%
Restaurant | Corporate and Other
Operating group information
Revenue, net, percentage	0.00%	0.00%
Wholesale
Operating group information
Net sales	$ 101,926	$ 94,376
Revenue, net, percentage	36.00%	35.00%
Wholesale | Operating Groups | Tommy Bahama
Operating group information
Revenue, net, percentage	24.00%	24.00%
Wholesale | Operating Groups | Lilly Pulitzer
Operating group information
Revenue, net, percentage	33.00%	32.00%
Wholesale | Operating Groups | Lanier Apparel
Operating group information
Revenue, net, percentage	99.00%	100.00%
Wholesale | Operating Groups | Southern Tide
Operating group information
Revenue, net, percentage	87.00%	88.00%
Wholesale | Corporate and Other
Operating group information
Revenue, net, percentage	30.00%	23.00%
Other
Operating group information
Net sales	$ 451	$ 702
Revenue, net, percentage	0.00%	0.00%
Other | Operating Groups | Tommy Bahama
Operating group information
Revenue, net, percentage	0.00%	0.00%
Other | Operating Groups | Lilly Pulitzer
Operating group information
Revenue, net, percentage	0.00%	0.00%
Other | Operating Groups | Lanier Apparel
Operating group information
Revenue, net, percentage	0.00%	0.00%
Other | Operating Groups | Southern Tide
Operating group information
Revenue, net, percentage	0.00%	0.00%
Other | Corporate and Other
Operating group information
Revenue, net, percentage	9.00%	17.00%