Intangible Assets and Goodwill (Details) - USD ($) $ in Thousands	12 Months Ended
	Feb. 03, 2018	Jan. 28, 2017	Jan. 30, 2016	Feb. 03, 2018	Jan. 28, 2017
Intangible assets with finite lives, which primarily consist of customer relationships:
Intangible assets with finite lives				$ 52,470	$ 46,030
Accumulated amortization				(38,612)	(35,785)
Total intangible assets with finite lives, net				13,858	10,245
Intangible assets with indefinite lives:
Trademarks				165,000	165,000
Total intangible assets, net	$ 175,245	$ 143,738	$ 146,134	178,858	175,245
Changes in carrying amount of intangible assets
Balance at the beginning of the period	175,245	143,738	146,134
Acquisition	5,940	33,377
Amortization	(2,404)	(2,150)	(1,926)
Other, including foreign currency	77	280	(470)
Balance at the end of the period	178,858	175,245	143,738
Expected amortization expense
2018				2,600
2019				1,200
2020				1,200
2021				1,000
2022				800
Operating Segments | Tommy Bahama
Intangible assets with indefinite lives:
Total intangible assets, net	113,625	114,944	117,102	112,157	113,625
Changes in carrying amount of intangible assets
Balance at the beginning of the period	113,625	114,944	117,102
Amortization	(1,580)	(1,599)	(1,688)
Other, including foreign currency	112	280	(470)
Balance at the end of the period	112,157	113,625	114,944
Operating Segments | Lilly Pulitzer
Intangible assets with indefinite lives:
Total intangible assets, net	28,595	28,794	29,032	29,749	28,595
Changes in carrying amount of intangible assets
Balance at the beginning of the period	28,595	28,794	29,032
Acquisition	1,500	0
Amortization	(346)	(199)	(238)
Other, including foreign currency	0	0	0
Balance at the end of the period	29,749	28,595	28,794
Operating Segments | Lanier Apparel
Intangible assets with indefinite lives:
Total intangible assets, net	3,048	3,048		2,841	3,048
Changes in carrying amount of intangible assets
Balance at the beginning of the period	3,048
Acquisition	0	3,137
Amortization	(172)	(89)
Other, including foreign currency	(35)	0	0
Balance at the end of the period	2,841	3,048
Operating Segments | Southern Tide
Intangible assets with indefinite lives:
Total intangible assets, net	29,977	29,977		29,689	$ 29,977
Changes in carrying amount of intangible assets
Balance at the beginning of the period	29,977
Acquisition	0	30,240
Amortization	(288)	(263)
Other, including foreign currency	0	0	0
Balance at the end of the period	29,689	29,977
Corporate and Other
Intangible assets with indefinite lives:
Total intangible assets, net	4,422			$ 4,422
Changes in carrying amount of intangible assets
Acquisition	4,440	0
Amortization	(18)
Other, including foreign currency	0	$ 0	$ 0
Balance at the end of the period	$ 4,422