Intangible Assets and Goodwill (Details) - USD ($) $ in Thousands	12 Months Ended
	Feb. 02, 2019	Feb. 03, 2018	Jan. 28, 2017	Feb. 02, 2019	Feb. 03, 2018
Intangible assets with finite lives, which primarily consist of customer relationships:
Intangible assets with finite lives				$ 51,929	$ 52,470
Accumulated amortization				(40,753)	(38,612)
Total intangible assets with finite lives, net				11,176	13,858
Intangible assets with indefinite lives:
Total intangible assets, net	$ 178,858	$ 175,245	$ 143,738	176,176	178,858
Changes in carrying amount of intangible assets
Balance at the beginning of the period	178,858	175,245	143,738
Acquisition	0	5,940	33,377
Amortization	(2,610)	(2,404)	(2,150)
Other, including foreign currency	(72)	77	280
Balance at the end of the period	176,176	178,858	175,245
Expected amortization expense
2018				1,200
2019				1,200
2020				1,000
2021				1,000
2022				800
Operating Segments | Tommy Bahama
Intangible assets with indefinite lives:
Total intangible assets, net	112,157	113,625	114,944	110,700	112,157
Changes in carrying amount of intangible assets
Balance at the beginning of the period	112,157	113,625	114,944
Acquisition	0	0	0
Amortization	(1,385)	(1,580)	(1,599)
Other, including foreign currency	(72)	112	280
Balance at the end of the period	110,700	112,157	113,625
Operating Segments | Lilly Pulitzer
Intangible assets with indefinite lives:
Total intangible assets, net	29,749	28,595	28,794	29,216	29,749
Changes in carrying amount of intangible assets
Balance at the beginning of the period	29,749	28,595	28,794
Acquisition	0	1,500	0
Amortization	(533)	(346)	(199)
Other, including foreign currency	0	0	0
Balance at the end of the period	29,216	29,749	28,595
Operating Segments | Lanier Apparel
Intangible assets with indefinite lives:
Total intangible assets, net	2,841	3,048	0	2,670	2,841
Changes in carrying amount of intangible assets
Balance at the beginning of the period	2,841	3,048	0
Acquisition	0	0	3,137
Amortization	(171)	(172)	(89)
Other, including foreign currency	0	(35)	0
Balance at the end of the period	2,670	2,841	3,048
Operating Segments | Southern Tide
Intangible assets with indefinite lives:
Total intangible assets, net	29,689	29,977	0	29,401	29,689
Changes in carrying amount of intangible assets
Balance at the beginning of the period	29,689	29,977	0
Acquisition	0	0	30,240
Amortization	(288)	(288)	(263)
Other, including foreign currency	0	0	0
Balance at the end of the period	29,401	29,689	29,977
Corporate and Other
Intangible assets with indefinite lives:
Total intangible assets, net	4,422	0	0	4,189	4,422
Changes in carrying amount of intangible assets
Balance at the beginning of the period	4,422	0	0
Acquisition	0	4,440	0
Amortization	(233)	(18)	0
Other, including foreign currency	0	0	0
Balance at the end of the period	$ 4,189	$ 4,422	$ 0
Tommy Bahama Trademarks
Intangible assets with indefinite lives:
Intangible assets with indefinite lives				110,700	110,700
Lilly Pulitzer Trademarks
Intangible assets with indefinite lives:
Intangible assets with indefinite lives				27,500	27,500
Southern Tide Trademarks
Intangible assets with indefinite lives:
Intangible assets with indefinite lives				$ 26,800	$ 26,800