<?xml version="1.0" encoding="us-ascii"?><InstanceReport xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><Version>2.4.0.8</Version><ReportLongName>1020 - Disclosure - Restructuring and Asset Impairment</ReportLongName><DisplayLabelColumn>true</DisplayLabelColumn><ShowElementNames>false</ShowElementNames><RoundingOption /><HasEmbeddedReports>false</HasEmbeddedReports><Columns><Column FlagID="0"><Id>1</Id><IsAbstractGroupTitle>false</IsAbstractGroupTitle><LabelSeparator>

</LabelSeparator><CurrencyCode /><FootnoteIndexer /><hasSegments>false</hasSegments><hasScenarios>false</hasScenarios><MCU><KeyName /><CurrencySymbol /><contextRef><ContextID>D2013Q2YTD</ContextID><EntitySchema>http://www.sec.gov/CIK</EntitySchema><EntityValue>0000837173</EntityValue><PeriodDisplayName /><PeriodType>duration</PeriodType><PeriodStartDate>2013-01-01T00:00:00</PeriodStartDate><PeriodEndDate>2013-06-30T00:00:00</PeriodEndDate><Segments /><Scenarios /></contextRef><UPS /><CurrencyCode /><OriginalCurrencyCode /></MCU><CurrencySymbol /><Labels><Label Key="CalendarSupplement" Id="0" Label="6 Months Ended" /><Label Key="Calendar" Id="1" Label="Jun. 30, 2013" /></Labels></Column></Columns><Rows><Row FlagID="0"><Id>1</Id><IsAbstractGroupTitle>true</IsAbstractGroupTitle><LabelSeparator>

</LabelSeparator><Level>1</Level><ElementName>us-gaap_RestructuringCostsAndAssetImpairmentChargesAbstract</ElementName><ElementPrefix>us-gaap_</ElementPrefix><IsBaseElement>true</IsBaseElement><BalanceType>na</BalanceType><PeriodType>duration</PeriodType><IsReportTitle>false</IsReportTitle><IsSegmentTitle>false</IsSegmentTitle><IsCalendarTitle>false</IsCalendarTitle><IsEquityPrevioslyReportedAsRow>false</IsEquityPrevioslyReportedAsRow><IsEquityAdjustmentRow>false</IsEquityAdjustmentRow><IsBeginningBalance>false</IsBeginningBalance><IsEndingBalance>false</IsEndingBalance><IsReverseSign>false</IsReverseSign><FootnoteIndexer /><Cells><Cell FlagID="0" ContextID="" UnitID=""><Id>1</Id><IsNumeric>false</IsNumeric><IsRatio>false</IsRatio><DisplayZeroAsNone>false</DisplayZeroAsNone><NumericAmount>0</NumericAmount><RoundedNumericAmount>0</RoundedNumericAmount><NonNumbericText /><FootnoteIndexer /><CurrencyCode /><CurrencySymbol /><IsIndependantCurrency>false</IsIndependantCurrency><ShowCurrencySymbol>false</ShowCurrencySymbol><DisplayDateInUSFormat>false</DisplayDateInUSFormat></Cell></Cells><ElementDataType>xbrli:stringItemType</ElementDataType><SimpleDataType>string</SimpleDataType><IsTotalLabel>false</IsTotalLabel><UnitID>0</UnitID><Label>Restructuring and Asset Impairment</Label></Row><Row FlagID="0"><Id>2</Id><IsAbstractGroupTitle>false</IsAbstractGroupTitle><LabelSeparator>

</LabelSeparator><Level>2</Level><ElementName>us-gaap_RestructuringImpairmentAndOtherActivitiesDisclosureTextBlock</ElementName><ElementPrefix>us-gaap_</ElementPrefix><IsBaseElement>true</IsBaseElement><BalanceType>na</BalanceType><PeriodType>duration</PeriodType><IsReportTitle>false</IsReportTitle><IsSegmentTitle>false</IsSegmentTitle><IsCalendarTitle>false</IsCalendarTitle><IsEquityPrevioslyReportedAsRow>false</IsEquityPrevioslyReportedAsRow><IsEquityAdjustmentRow>false</IsEquityAdjustmentRow><IsBeginningBalance>false</IsBeginningBalance><IsEndingBalance>false</IsEndingBalance><IsReverseSign>false</IsReverseSign><PreferredLabelRole>terseLabel</PreferredLabelRole><FootnoteIndexer /><Cells><Cell FlagID="0" ContextID="D2013Q2YTD" UnitID=""><Id>1</Id><IsNumeric>false</IsNumeric><IsRatio>false</IsRatio><DisplayZeroAsNone>false</DisplayZeroAsNone><NumericAmount>0</NumericAmount><RoundedNumericAmount>0</RoundedNumericAmount><NonNumbericText>&lt;div style="font-size:10.0pt;font-family:Times New Roman;"&gt;
&lt;p style="FONT-FAMILY: times;"&gt;&lt;font size="2"&gt;&lt;b&gt;Note&amp;#160;2&amp;#8212;Restructuring and Asset Impairment&lt;/b&gt;&lt;/font&gt;&lt;/p&gt;
&lt;p style="FONT-FAMILY: times;"&gt;&lt;font size="2"&gt;&amp;#160;&amp;#160;&amp;#160;&amp;#160;&amp;#160;&amp;#160;&amp;#160;&amp;#160;In response to the current depressed price environment for thermal coal and geological issues within the Company's North River mine, the Company announced plans to close the mine in 2013 approximately nine months earlier than the previously expected end of mine life of 2014. In connection with the accelerated closure, the Company renegotiated an unfavorably priced coal sales contract in the second quarter of 2013 to reduce the total tons committed for sale from this mine. The renegotiation will allow a closure of this mine in the latter half of 2013. For the three and six months ended June&amp;#160;30, 2013 the Company recognized a gain of approximately $17.0&amp;#160;million due to the release of the related below market contract liability that was obtained through the acquisition of North River. The benefit of $17.0&amp;#160;million was partially offset by asset impairment charges of approximately $8.0&amp;#160;million, all related to the accelerated closure of the North River Mine. The Company also incurred $3.3&amp;#160;million and $10.7&amp;#160;million of costs related to the curtailment of the Willow Creek mine for the three and six months ended June&amp;#160;30, 2013, respectively.&lt;/font&gt;&lt;/p&gt;
&lt;/div&gt;</NonNumbericText><FootnoteIndexer /><CurrencyCode /><CurrencySymbol /><IsIndependantCurrency>false</IsIndependantCurrency><ShowCurrencySymbol>false</ShowCurrencySymbol><DisplayDateInUSFormat>false</DisplayDateInUSFormat></Cell></Cells><ElementDataType>nonnum:textBlockItemType</ElementDataType><SimpleDataType>na</SimpleDataType><ElementDefenition>The entire disclosure for aggregate costs incurred during the period for restructuring, remediation, implementation, integration and asset impairment activities. Description of restructuring activities such as exit and disposal activities, could include facts and circumstances leading to the plan, the expected plan completion date, the major types of costs associated with the plan activities, total expected costs, the accrual balance at the end of the period, and the periods over which the remaining accrual will be settled. This would also include disclosure of the charge against earnings resulting from the aggregate write down of all assets from their carrying value to their fair value.</ElementDefenition><ElementReferences>No definition available.</ElementReferences><IsTotalLabel>false</IsTotalLabel><UnitID>0</UnitID><Label>Restructuring and Asset Impairment</Label></Row></Rows><Footnotes /><IsEquityReport>false</IsEquityReport><ReportName>Restructuring and Asset Impairment</ReportName><MonetaryRoundingLevel>UnKnown</MonetaryRoundingLevel><SharesRoundingLevel>UnKnown</SharesRoundingLevel><PerShareRoundingLevel>UnKnown</PerShareRoundingLevel><ExchangeRateRoundingLevel>UnKnown</ExchangeRateRoundingLevel><HasCustomUnits>true</HasCustomUnits><IsEmbedReport>false</IsEmbedReport><IsMultiCurrency>false</IsMultiCurrency><ReportType>Sheet</ReportType><RoleURI>http://www.walterenergy.com/role/DisclosureRestructuringAndAssetImpairment</RoleURI><NumberOfCols>1</NumberOfCols><NumberOfRows>2</NumberOfRows></InstanceReport>
