Intangible Assets and Goodwill - By operating group (Details) - USD ($) $ in Thousands	12 Months Ended
	Feb. 01, 2020	Feb. 02, 2019	Feb. 03, 2018
Changes in carrying amount of intangible assets
Balance at the beginning of the period	$ 176,176	$ 178,858	$ 175,245
Acquisition			5,940
Amortization	(1,171)	(2,610)	(2,404)
Other, including foreign currency		(72)	77
Balance at the end of the period	175,005	176,176	178,858
Operating Segments | Tommy Bahama
Changes in carrying amount of intangible assets
Balance at the beginning of the period	110,700	112,157	113,625
Amortization		(1,385)	(1,580)
Other, including foreign currency		(72)	112
Balance at the end of the period	110,700	110,700	112,157
Operating Segments | Lilly Pulitzer
Changes in carrying amount of intangible assets
Balance at the beginning of the period	29,216	29,749	28,595
Acquisition			1,500
Amortization	(475)	(533)	(346)
Balance at the end of the period	28,741	29,216	29,749
Operating Segments | Lanier Apparel
Changes in carrying amount of intangible assets
Balance at the beginning of the period	2,670	2,841	3,048
Amortization	(171)	(171)	(172)
Other, including foreign currency			(35)
Balance at the end of the period	2,499	2,670	2,841
Operating Segments | Southern Tide
Changes in carrying amount of intangible assets
Balance at the beginning of the period	29,401	29,689	29,977
Amortization	(291)	(288)	(288)
Balance at the end of the period	29,110	29,401	29,689
Corporate and Other
Changes in carrying amount of intangible assets
Balance at the beginning of the period	4,189	4,422
Acquisition			4,440
Amortization	(234)	(233)	(18)
Balance at the end of the period	$ 3,955	$ 4,189	$ 4,422