Intangible Assets and Goodwill - Goodwill (Details) - USD ($) $ in Thousands	12 Months Ended
	Feb. 01, 2020	Feb. 02, 2019	Feb. 03, 2018
Changes in the carrying amount of goodwill
Balance at the beginning of the period	$ 66,621	$ 66,703	$ 60,015
Other, including foreign currency	(43)	(82)	46
Acquisition			6,642
Balance at the end of the period	66,578	66,621	66,703
Operating Segments | Tommy Bahama
Changes in the carrying amount of goodwill
Balance at the beginning of the period	754	821	775
Other, including foreign currency	(43)	(67)	46
Balance at the end of the period	711	754	821
Operating Segments | Lilly Pulitzer
Changes in the carrying amount of goodwill
Balance at the beginning of the period	19,522	19,522	16,495
Acquisition			3,027
Balance at the end of the period	19,522	19,522	19,522
Operating Segments | Southern Tide
Changes in the carrying amount of goodwill
Balance at the beginning of the period	42,745	42,745	42,745
Balance at the end of the period	42,745	42,745	42,745
Corporate and Other
Changes in the carrying amount of goodwill
Balance at the beginning of the period	3,600	3,615
Other, including foreign currency		(15)
Acquisition			3,615
Balance at the end of the period	$ 3,600	$ 3,600	$ 3,615