<?xml version="1.0" encoding="us-ascii"?><InstanceReport xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><Version>2.4.0.8</Version><ReportLongName>4040 - Disclosure - Asset Impairment and Store Closing Charges (Details)</ReportLongName><DisplayLabelColumn>true</DisplayLabelColumn><ShowElementNames>false</ShowElementNames><RoundingOption>In Thousands, unless otherwise specified</RoundingOption><HasEmbeddedReports>false</HasEmbeddedReports><Columns><Column FlagID="0"><Id>1</Id><IsAbstractGroupTitle>false</IsAbstractGroupTitle><LabelSeparator>

</LabelSeparator><CurrencyCode>USD</CurrencyCode><FootnoteIndexer /><hasSegments>false</hasSegments><hasScenarios>false</hasScenarios><MCU><KeyName /><CurrencySymbol>$</CurrencySymbol><contextRef><ContextID>D2013Q2YTD</ContextID><EntitySchema>http://www.sec.gov/CIK</EntitySchema><EntityValue>0000028917</EntityValue><PeriodDisplayName /><PeriodType>duration</PeriodType><PeriodStartDate>2013-02-03T00:00:00</PeriodStartDate><PeriodEndDate>2013-08-03T00:00:00</PeriodEndDate><Segments /><Scenarios /></contextRef><UPS><UnitProperty><UnitID>USD</UnitID><UnitType>Standard</UnitType><StandardMeasure><MeasureSchema>http://www.xbrl.org/2003/iso4217</MeasureSchema><MeasureValue>USD</MeasureValue><MeasureNamespace>iso4217</MeasureNamespace><MeasureDescription /><MeasureType /></StandardMeasure><Scale>0</Scale></UnitProperty></UPS><CurrencyCode>USD</CurrencyCode><OriginalCurrencyCode>USD</OriginalCurrencyCode></MCU><CurrencySymbol>$</CurrencySymbol><Labels><Label Key="CalendarSupplement" Id="0" Label="6 Months Ended" /><Label Key="Calendar" Id="1" Label="Aug. 03, 2013" /></Labels></Column></Columns><Rows><Row FlagID="0"><Id>1</Id><IsAbstractGroupTitle>true</IsAbstractGroupTitle><LabelSeparator>

</LabelSeparator><Level>1</Level><ElementName>dds_AssetImpairmentAndStoreClosingChargesDisclosureAbstract</ElementName><ElementPrefix>dds_</ElementPrefix><IsBaseElement>false</IsBaseElement><BalanceType>na</BalanceType><PeriodType>duration</PeriodType><IsReportTitle>false</IsReportTitle><IsSegmentTitle>false</IsSegmentTitle><IsCalendarTitle>false</IsCalendarTitle><IsEquityPrevioslyReportedAsRow>false</IsEquityPrevioslyReportedAsRow><IsEquityAdjustmentRow>false</IsEquityAdjustmentRow><IsBeginningBalance>false</IsBeginningBalance><IsEndingBalance>false</IsEndingBalance><IsReverseSign>false</IsReverseSign><FootnoteIndexer /><Cells><Cell FlagID="0" ContextID="" UnitID=""><Id>1</Id><IsNumeric>false</IsNumeric><IsRatio>false</IsRatio><DisplayZeroAsNone>false</DisplayZeroAsNone><NumericAmount>0</NumericAmount><RoundedNumericAmount>0</RoundedNumericAmount><NonNumbericText /><FootnoteIndexer /><CurrencyCode /><CurrencySymbol /><IsIndependantCurrency>false</IsIndependantCurrency><ShowCurrencySymbol>false</ShowCurrencySymbol><DisplayDateInUSFormat>false</DisplayDateInUSFormat></Cell></Cells><ElementDataType>xbrli:stringItemType</ElementDataType><SimpleDataType>string</SimpleDataType><IsTotalLabel>false</IsTotalLabel><UnitID>0</UnitID><Label>Asset Impairment and Store Closing Charges</Label></Row><Row FlagID="0"><Id>2</Id><IsAbstractGroupTitle>false</IsAbstractGroupTitle><LabelSeparator>

</LabelSeparator><Level>2</Level><ElementName>us-gaap_RestructuringSettlementAndImpairmentProvisions</ElementName><ElementPrefix>us-gaap_</ElementPrefix><IsBaseElement>true</IsBaseElement><BalanceType>debit</BalanceType><PeriodType>duration</PeriodType><IsReportTitle>false</IsReportTitle><IsSegmentTitle>false</IsSegmentTitle><IsCalendarTitle>false</IsCalendarTitle><IsEquityPrevioslyReportedAsRow>false</IsEquityPrevioslyReportedAsRow><IsEquityAdjustmentRow>false</IsEquityAdjustmentRow><IsBeginningBalance>false</IsBeginningBalance><IsEndingBalance>false</IsEndingBalance><IsReverseSign>false</IsReverseSign><PreferredLabelRole>verboseLabel</PreferredLabelRole><FootnoteIndexer /><Cells><Cell FlagID="0" ContextID="D2013Q2YTD" UnitID="USD"><Id>1</Id><IsNumeric>true</IsNumeric><IsRatio>false</IsRatio><DisplayZeroAsNone>false</DisplayZeroAsNone><NumericAmount>6527000</NumericAmount><RoundedNumericAmount>6527</RoundedNumericAmount><NonNumbericText /><FootnoteIndexer /><CurrencyCode>USD</CurrencyCode><CurrencySymbol>$</CurrencySymbol><IsIndependantCurrency>false</IsIndependantCurrency><ShowCurrencySymbol>true</ShowCurrencySymbol><DisplayDateInUSFormat>false</DisplayDateInUSFormat></Cell></Cells><ElementDataType>xbrli:monetaryItemType</ElementDataType><SimpleDataType>monetary</SimpleDataType><ElementDefenition>The aggregate amount provided for estimated restructuring charges, remediation costs, and asset impairment loss during an accounting period. Generally, these items are either unusual or infrequent, but not both (in which case they would be extraordinary items).</ElementDefenition><ElementReferences>Reference 1: http://www.xbrl.org/2003/role/presentationRef

 -Publisher FASB

 -Name Accounting Standards Codification

 -Topic 225

 -SubTopic 10

 -Section S99

 -Paragraph 2

 -Subparagraph (SX 210.5-03.3)

 -URI http://asc.fasb.org/extlink&amp;oid=6880815&amp;loc=d3e20235-122688



</ElementReferences><IsTotalLabel>false</IsTotalLabel><UnitID>2</UnitID><Label>Asset impairment and store closing charges</Label></Row><Row FlagID="0"><Id>3</Id><IsAbstractGroupTitle>true</IsAbstractGroupTitle><LabelSeparator>

</LabelSeparator><Level>2</Level><ElementName>us-gaap_RestructuringReserveRollForward</ElementName><ElementPrefix>us-gaap_</ElementPrefix><IsBaseElement>true</IsBaseElement><BalanceType>na</BalanceType><PeriodType>duration</PeriodType><IsReportTitle>false</IsReportTitle><IsSegmentTitle>false</IsSegmentTitle><IsCalendarTitle>false</IsCalendarTitle><IsEquityPrevioslyReportedAsRow>false</IsEquityPrevioslyReportedAsRow><IsEquityAdjustmentRow>false</IsEquityAdjustmentRow><IsBeginningBalance>false</IsBeginningBalance><IsEndingBalance>false</IsEndingBalance><IsReverseSign>false</IsReverseSign><PreferredLabelRole>terseLabel</PreferredLabelRole><FootnoteIndexer /><Cells><Cell FlagID="0" ContextID="" UnitID=""><Id>1</Id><IsNumeric>false</IsNumeric><IsRatio>false</IsRatio><DisplayZeroAsNone>false</DisplayZeroAsNone><NumericAmount>0</NumericAmount><RoundedNumericAmount>0</RoundedNumericAmount><NonNumbericText /><FootnoteIndexer /><CurrencyCode /><CurrencySymbol /><IsIndependantCurrency>false</IsIndependantCurrency><ShowCurrencySymbol>false</ShowCurrencySymbol><DisplayDateInUSFormat>false</DisplayDateInUSFormat></Cell></Cells><ElementDataType>xbrli:stringItemType</ElementDataType><SimpleDataType>string</SimpleDataType><IsTotalLabel>false</IsTotalLabel><UnitID>0</UnitID><Label>Summary of activity in reserve established for store closing charges</Label></Row><Row FlagID="0"><Id>4</Id><IsAbstractGroupTitle>false</IsAbstractGroupTitle><LabelSeparator>

</LabelSeparator><Level>3</Level><ElementName>us-gaap_RestructuringReserve</ElementName><ElementPrefix>us-gaap_</ElementPrefix><IsBaseElement>true</IsBaseElement><BalanceType>credit</BalanceType><PeriodType>instant</PeriodType><IsReportTitle>false</IsReportTitle><IsSegmentTitle>false</IsSegmentTitle><IsCalendarTitle>false</IsCalendarTitle><IsEquityPrevioslyReportedAsRow>false</IsEquityPrevioslyReportedAsRow><IsEquityAdjustmentRow>false</IsEquityAdjustmentRow><IsBeginningBalance>true</IsBeginningBalance><IsEndingBalance>false</IsEndingBalance><IsReverseSign>false</IsReverseSign><PreferredLabelRole>periodStartLabel</PreferredLabelRole><FootnoteIndexer /><Cells><Cell FlagID="0" ContextID="I2012" UnitID="USD"><Id>1</Id><IsNumeric>true</IsNumeric><IsRatio>false</IsRatio><DisplayZeroAsNone>false</DisplayZeroAsNone><NumericAmount>251000</NumericAmount><RoundedNumericAmount>251</RoundedNumericAmount><NonNumbericText /><FootnoteIndexer /><CurrencyCode /><CurrencySymbol /><IsIndependantCurrency>false</IsIndependantCurrency><ShowCurrencySymbol>false</ShowCurrencySymbol><DisplayDateInUSFormat>false</DisplayDateInUSFormat></Cell></Cells><ElementDataType>xbrli:monetaryItemType</ElementDataType><SimpleDataType>monetary</SimpleDataType><ElementDefenition>Carrying amount (including both current and noncurrent portions of the accrual) as of the balance sheet date pertaining to a specified type of cost associated with exit from or disposal of business activities or restructuring pursuant to a duly authorized plan.</ElementDefenition><ElementReferences>Reference 1: http://www.xbrl.org/2003/role/presentationRef

 -Publisher FASB

 -Name Emerging Issues Task Force (EITF)

 -Number 95-3

 -LegacyDoc This reference is SUPERSEDED by the Accounting Standards Codification effective for interim and annual periods ending after September 15, 2009.  This reference is included to help users transition from the previous accounting hierarchy and will be removed from future versions of this taxonomy.



Reference 2: http://www.xbrl.org/2003/role/presentationRef

 -Publisher FASB

 -Name Accounting Standards Codification

 -Topic 420

 -SubTopic 10

 -Section S99

 -Paragraph 2

 -Subparagraph (SAB TOPIC 5.P.4(b))

 -URI http://asc.fasb.org/extlink&amp;oid=6394695&amp;loc=d3e140904-122747



Reference 3: http://www.xbrl.org/2003/role/presentationRef

 -Publisher SEC

 -Name Staff Accounting Bulletin (SAB)

 -Number Topic 5

 -Section P

 -Subsection 3, 4



</ElementReferences><IsTotalLabel>false</IsTotalLabel><UnitID>2</UnitID><Label>Rentals, property taxes and utilities, Balance Beginning of Period</Label></Row><Row FlagID="0"><Id>5</Id><IsAbstractGroupTitle>false</IsAbstractGroupTitle><LabelSeparator>

</LabelSeparator><Level>3</Level><ElementName>us-gaap_RestructuringCharges</ElementName><ElementPrefix>us-gaap_</ElementPrefix><IsBaseElement>true</IsBaseElement><BalanceType>debit</BalanceType><PeriodType>duration</PeriodType><IsReportTitle>false</IsReportTitle><IsSegmentTitle>false</IsSegmentTitle><IsCalendarTitle>false</IsCalendarTitle><IsEquityPrevioslyReportedAsRow>false</IsEquityPrevioslyReportedAsRow><IsEquityAdjustmentRow>false</IsEquityAdjustmentRow><IsBeginningBalance>false</IsBeginningBalance><IsEndingBalance>false</IsEndingBalance><IsReverseSign>false</IsReverseSign><PreferredLabelRole>terseLabel</PreferredLabelRole><FootnoteIndexer /><Cells><Cell FlagID="0" ContextID="D2013Q2YTD" UnitID="USD"><Id>1</Id><IsNumeric>true</IsNumeric><IsRatio>false</IsRatio><DisplayZeroAsNone>false</DisplayZeroAsNone><NumericAmount>101000</NumericAmount><RoundedNumericAmount>101</RoundedNumericAmount><NonNumbericText /><FootnoteIndexer /><CurrencyCode /><CurrencySymbol /><IsIndependantCurrency>false</IsIndependantCurrency><ShowCurrencySymbol>false</ShowCurrencySymbol><DisplayDateInUSFormat>false</DisplayDateInUSFormat></Cell></Cells><ElementDataType>xbrli:monetaryItemType</ElementDataType><SimpleDataType>monetary</SimpleDataType><ElementDefenition>Amount charged against earnings in the period for incurred and estimated costs associated with exit from or disposal of business activities or restructurings pursuant to a duly authorized plan, excluding asset retirement obligations.</ElementDefenition><ElementReferences>Reference 1: http://www.xbrl.org/2003/role/presentationRef

 -Publisher FASB

 -Name Accounting Standards Codification

 -Topic 230

 -SubTopic 10

 -Section 45

 -Paragraph 28

 -Subparagraph (b)

 -URI http://asc.fasb.org/extlink&amp;oid=6943989&amp;loc=d3e3602-108585



Reference 2: http://www.xbrl.org/2003/role/presentationRef

 -Publisher FASB

 -Name Accounting Standards Codification

 -Topic 420

 -SubTopic 10

 -Section 50

 -Paragraph 1

 -Subparagraph (b)

 -URI http://asc.fasb.org/extlink&amp;oid=6394359&amp;loc=d3e17939-110869



Reference 3: http://www.xbrl.org/2003/role/presentationRef

 -Publisher FASB

 -Name Accounting Standards Codification

 -Topic 420

 -SubTopic 10

 -Section S99

 -Paragraph 2

 -Subparagraph (SAB TOPIC 5.P.4(b))

 -URI http://asc.fasb.org/extlink&amp;oid=6394695&amp;loc=d3e140904-122747



Reference 4: http://www.xbrl.org/2003/role/presentationRef

 -Publisher FASB

 -Name Accounting Standards Codification

 -Topic 420

 -SubTopic 10

 -Section S99

 -Paragraph 1

 -Subparagraph (SAB TOPIC 5.P.3)

 -URI http://asc.fasb.org/extlink&amp;oid=6394695&amp;loc=d3e140864-122747



Reference 5: http://www.xbrl.org/2003/role/presentationRef

 -Publisher SEC

 -Name Staff Accounting Bulletin (SAB)

 -Number Topic 5

 -Section P

 -Subsection 3, 4



</ElementReferences><IsTotalLabel>false</IsTotalLabel><UnitID>2</UnitID><Label>Adjustments and Charges</Label></Row><Row FlagID="0"><Id>6</Id><IsAbstractGroupTitle>false</IsAbstractGroupTitle><LabelSeparator>

</LabelSeparator><Level>3</Level><ElementName>us-gaap_RestructuringReserveSettledWithCash</ElementName><ElementPrefix>us-gaap_</ElementPrefix><IsBaseElement>true</IsBaseElement><BalanceType>debit</BalanceType><PeriodType>duration</PeriodType><IsReportTitle>false</IsReportTitle><IsSegmentTitle>false</IsSegmentTitle><IsCalendarTitle>false</IsCalendarTitle><IsEquityPrevioslyReportedAsRow>false</IsEquityPrevioslyReportedAsRow><IsEquityAdjustmentRow>false</IsEquityAdjustmentRow><IsBeginningBalance>false</IsBeginningBalance><IsEndingBalance>false</IsEndingBalance><IsReverseSign>false</IsReverseSign><PreferredLabelRole>terseLabel</PreferredLabelRole><FootnoteIndexer /><Cells><Cell FlagID="0" ContextID="D2013Q2YTD" UnitID="USD"><Id>1</Id><IsNumeric>true</IsNumeric><IsRatio>false</IsRatio><DisplayZeroAsNone>false</DisplayZeroAsNone><NumericAmount>224000</NumericAmount><RoundedNumericAmount>224</RoundedNumericAmount><NonNumbericText /><FootnoteIndexer /><CurrencyCode /><CurrencySymbol /><IsIndependantCurrency>false</IsIndependantCurrency><ShowCurrencySymbol>false</ShowCurrencySymbol><DisplayDateInUSFormat>false</DisplayDateInUSFormat></Cell></Cells><ElementDataType>xbrli:monetaryItemType</ElementDataType><SimpleDataType>monetary</SimpleDataType><ElementDefenition>Amount of cash paid in the period to fully or partially settle a specified, previously accrued type of restructuring cost.</ElementDefenition><ElementReferences>Reference 1: http://www.xbrl.org/2003/role/presentationRef

 -Publisher FASB

 -Name Emerging Issues Task Force (EITF)

 -Number 95-3

 -LegacyDoc This reference is SUPERSEDED by the Accounting Standards Codification effective for interim and annual periods ending after September 15, 2009.  This reference is included to help users transition from the previous accounting hierarchy and will be removed from future versions of this taxonomy.



Reference 2: http://www.xbrl.org/2003/role/presentationRef

 -Publisher SEC

 -Name Staff Accounting Bulletin (SAB)

 -Number Topic 5

 -Section P

 -Subsection 3, 4



Reference 3: http://www.xbrl.org/2003/role/presentationRef

 -Publisher FASB

 -Name Statement of Financial Accounting Standard (FAS)

 -Number 146

 -LegacyDoc This reference is SUPERSEDED by the Accounting Standards Codification effective for interim and annual periods ending after September 15, 2009.  This reference is included to help users transition from the previous accounting hierarchy and will be removed from future versions of this taxonomy.



Reference 4: http://www.xbrl.org/2003/role/presentationRef

 -Publisher FASB

 -Name Accounting Standards Codification

 -Topic 420

 -SubTopic 10

 -Section S99

 -Paragraph 2

 -Subparagraph (SAB TOPIC 5.P.4(b)(2))

 -URI http://asc.fasb.org/extlink&amp;oid=6394695&amp;loc=d3e140904-122747



Reference 5: http://www.xbrl.org/2003/role/presentationRef

 -Publisher FASB

 -Name Accounting Standards Codification

 -Topic 420

 -SubTopic 10

 -Section 50

 -Paragraph 1

 -Subparagraph (b) (2)

 -URI http://asc.fasb.org/extlink&amp;oid=6394359&amp;loc=d3e17939-110869



</ElementReferences><IsTotalLabel>false</IsTotalLabel><UnitID>2</UnitID><Label>Cash Payments</Label></Row><Row FlagID="0"><Id>7</Id><IsAbstractGroupTitle>false</IsAbstractGroupTitle><LabelSeparator>

</LabelSeparator><Level>3</Level><ElementName>us-gaap_RestructuringReserve</ElementName><ElementPrefix>us-gaap_</ElementPrefix><IsBaseElement>true</IsBaseElement><BalanceType>credit</BalanceType><PeriodType>instant</PeriodType><IsReportTitle>false</IsReportTitle><IsSegmentTitle>false</IsSegmentTitle><IsCalendarTitle>false</IsCalendarTitle><IsEquityPrevioslyReportedAsRow>false</IsEquityPrevioslyReportedAsRow><IsEquityAdjustmentRow>false</IsEquityAdjustmentRow><IsBeginningBalance>false</IsBeginningBalance><IsEndingBalance>true</IsEndingBalance><IsReverseSign>false</IsReverseSign><PreferredLabelRole>periodEndLabel</PreferredLabelRole><FootnoteIndexer /><Cells><Cell FlagID="0" ContextID="I2013Q2" UnitID="USD"><Id>1</Id><IsNumeric>true</IsNumeric><IsRatio>false</IsRatio><DisplayZeroAsNone>false</DisplayZeroAsNone><NumericAmount>128000</NumericAmount><RoundedNumericAmount>128</RoundedNumericAmount><NonNumbericText /><FootnoteIndexer /><CurrencyCode>USD</CurrencyCode><CurrencySymbol>$</CurrencySymbol><IsIndependantCurrency>false</IsIndependantCurrency><ShowCurrencySymbol>true</ShowCurrencySymbol><DisplayDateInUSFormat>false</DisplayDateInUSFormat></Cell></Cells><ElementDataType>xbrli:monetaryItemType</ElementDataType><SimpleDataType>monetary</SimpleDataType><ElementDefenition>Carrying amount (including both current and noncurrent portions of the accrual) as of the balance sheet date pertaining to a specified type of cost associated with exit from or disposal of business activities or restructuring pursuant to a duly authorized plan.</ElementDefenition><ElementReferences>Reference 1: http://www.xbrl.org/2003/role/presentationRef

 -Publisher FASB

 -Name Emerging Issues Task Force (EITF)

 -Number 95-3

 -LegacyDoc This reference is SUPERSEDED by the Accounting Standards Codification effective for interim and annual periods ending after September 15, 2009.  This reference is included to help users transition from the previous accounting hierarchy and will be removed from future versions of this taxonomy.



Reference 2: http://www.xbrl.org/2003/role/presentationRef

 -Publisher FASB

 -Name Accounting Standards Codification

 -Topic 420

 -SubTopic 10

 -Section S99

 -Paragraph 2

 -Subparagraph (SAB TOPIC 5.P.4(b))

 -URI http://asc.fasb.org/extlink&amp;oid=6394695&amp;loc=d3e140904-122747



Reference 3: http://www.xbrl.org/2003/role/presentationRef

 -Publisher SEC

 -Name Staff Accounting Bulletin (SAB)

 -Number Topic 5

 -Section P

 -Subsection 3, 4



</ElementReferences><IsTotalLabel>false</IsTotalLabel><UnitID>2</UnitID><Label>Rentals, property taxes and utilities, Balance End of Period</Label></Row></Rows><Footnotes /><IsEquityReport>false</IsEquityReport><ReportName>Asset Impairment and Store Closing Charges (Details) (USD $)</ReportName><MonetaryRoundingLevel>Thousands</MonetaryRoundingLevel><SharesRoundingLevel>UnKnown</SharesRoundingLevel><PerShareRoundingLevel>UnKnown</PerShareRoundingLevel><ExchangeRateRoundingLevel>UnKnown</ExchangeRateRoundingLevel><HasCustomUnits>true</HasCustomUnits><IsEmbedReport>false</IsEmbedReport><IsMultiCurrency>false</IsMultiCurrency><ReportType>Sheet</ReportType><RoleURI>http://www.dillards.com/role/DisclosureAssetImpairmentAndStoreClosingChargesDetails</RoleURI><NumberOfCols>1</NumberOfCols><NumberOfRows>7</NumberOfRows></InstanceReport>
