<?xml version="1.0" encoding="us-ascii"?><InstanceReport xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><Version>2.4.0.8</Version><ReportLongName>008 - Disclosure - Note 4 - Loan Payable to Bank</ReportLongName><DisplayLabelColumn>true</DisplayLabelColumn><ShowElementNames>false</ShowElementNames><RoundingOption /><HasEmbeddedReports>false</HasEmbeddedReports><Columns><Column FlagID="0"><Id>1</Id><IsAbstractGroupTitle>false</IsAbstractGroupTitle><LabelSeparator>

</LabelSeparator><CurrencyCode /><FootnoteIndexer /><hasSegments>false</hasSegments><hasScenarios>false</hasScenarios><MCU><KeyName /><CurrencySymbol /><contextRef><ContextID>c0_From1Apr2013To30Jun2013</ContextID><EntitySchema>http://www.sec.gov/CIK</EntitySchema><EntityValue>0000892907</EntityValue><PeriodDisplayName /><PeriodType>duration</PeriodType><PeriodStartDate>2013-04-01T00:00:00</PeriodStartDate><PeriodEndDate>2013-06-30T00:00:00</PeriodEndDate><Segments /><Scenarios /></contextRef><UPS /><CurrencyCode /><OriginalCurrencyCode /></MCU><CurrencySymbol /><Labels><Label Key="CalendarSupplement" Id="0" Label="3 Months Ended" /><Label Key="Calendar" Id="1" Label="Jun. 30, 2013" /></Labels></Column></Columns><Rows><Row FlagID="0"><Id>1</Id><IsAbstractGroupTitle>true</IsAbstractGroupTitle><LabelSeparator>

</LabelSeparator><Level>1</Level><ElementName>us-gaap_DebtDisclosureAbstract</ElementName><ElementPrefix>us-gaap_</ElementPrefix><IsBaseElement>true</IsBaseElement><BalanceType>na</BalanceType><PeriodType>duration</PeriodType><IsReportTitle>false</IsReportTitle><IsSegmentTitle>false</IsSegmentTitle><IsCalendarTitle>false</IsCalendarTitle><IsEquityPrevioslyReportedAsRow>false</IsEquityPrevioslyReportedAsRow><IsEquityAdjustmentRow>false</IsEquityAdjustmentRow><IsBeginningBalance>false</IsBeginningBalance><IsEndingBalance>false</IsEndingBalance><IsReverseSign>false</IsReverseSign><FootnoteIndexer /><Cells><Cell FlagID="0" ContextID="" UnitID=""><Id>1</Id><IsNumeric>false</IsNumeric><IsRatio>false</IsRatio><DisplayZeroAsNone>false</DisplayZeroAsNone><NumericAmount>0</NumericAmount><RoundedNumericAmount>0</RoundedNumericAmount><NonNumbericText /><FootnoteIndexer /><CurrencyCode /><CurrencySymbol /><IsIndependantCurrency>false</IsIndependantCurrency><ShowCurrencySymbol>false</ShowCurrencySymbol><DisplayDateInUSFormat>false</DisplayDateInUSFormat></Cell></Cells><ElementDataType>xbrli:stringItemType</ElementDataType><SimpleDataType>string</SimpleDataType><IsTotalLabel>false</IsTotalLabel><UnitID>0</UnitID><Label>Debt Disclosure [Abstract]</Label></Row><Row FlagID="0"><Id>2</Id><IsAbstractGroupTitle>false</IsAbstractGroupTitle><LabelSeparator>

</LabelSeparator><Level>2</Level><ElementName>us-gaap_DebtDisclosureTextBlock</ElementName><ElementPrefix>us-gaap_</ElementPrefix><IsBaseElement>true</IsBaseElement><BalanceType>na</BalanceType><PeriodType>duration</PeriodType><IsReportTitle>false</IsReportTitle><IsSegmentTitle>false</IsSegmentTitle><IsCalendarTitle>false</IsCalendarTitle><IsEquityPrevioslyReportedAsRow>false</IsEquityPrevioslyReportedAsRow><IsEquityAdjustmentRow>false</IsEquityAdjustmentRow><IsBeginningBalance>false</IsBeginningBalance><IsEndingBalance>false</IsEndingBalance><IsReverseSign>false</IsReverseSign><PreferredLabelRole>terseLabel</PreferredLabelRole><FootnoteIndexer /><Cells><Cell FlagID="0" ContextID="c0_From1Apr2013To30Jun2013" UnitID=""><Id>1</Id><IsNumeric>false</IsNumeric><IsRatio>false</IsRatio><DisplayZeroAsNone>false</DisplayZeroAsNone><NumericAmount>0</NumericAmount><RoundedNumericAmount>0</RoundedNumericAmount><NonNumbericText>&lt;p style="TEXT-ALIGN: left; LINE-HEIGHT: 1.25; MARGIN: 0pt" id="PARA693"&gt;

      &lt;font style="FONT-FAMILY: Times New Roman, Times, serif; FONT-SIZE: 10pt"&gt;4.&amp;#160;&amp;#160;&amp;#160;&amp;#160;&amp;#160;&amp;#160;&amp;#160;&amp;#160;&amp;#160;

      &amp;#160;&amp;#160;&amp;#160;Loan Payable to Bank&lt;/font&gt;

    &lt;/p&gt;&lt;br/&gt;&lt;p style="TEXT-ALIGN: justify; LINE-HEIGHT: 1.25; TEXT-INDENT: 36pt; MARGIN: 0pt" id="PARA695"&gt;

      &lt;font style="FONT-FAMILY: Times New Roman, Times, serif; FONT-SIZE: 10pt"&gt;In

      August 2013, we entered into a second amendment to our

      October 2010 amended and restated credit facility with our

      existing bank, Bank of America, N.A. (the

      &amp;#8220;Lender&amp;#8221;), which was first amended in May 2012.

      The credit facility provides for advances up to $75.0 million

      (previously $65.0 million from January 1 of each year through

      August 31 of each year and $70.0 million from September 1 of

      each year through December 31 of each year). This facility

      provides for up to $10.0 million in authorized letters of

      credit within the $75.0 million facility. The amount we can

      borrow under this credit facility is limited to fixed

      percentages of the value of various categories of accounts

      receivable and fixed percentages of the value of various

      categories of eligible inventory, minus certain reserves. As

      amended by the second amendment, eligible inventory now

      includes food, rental equipment inventory up to $2.5 million

      of borrowing base, and ski lift ticket inventory from

      December 1 to March 31 of each year to the extent the lift

      tickets are subject to a guaranteed buy-back. Our borrowing

      capacity and thus the adequacy of our working capital would

      be adversely affected if we experienced a significant

      decrease in eligible inventory, whether due to our

      vendors&amp;#8217; unwillingness to ship us merchandise, the

      aging of inventory and/or an unfavorable inventory appraisal.

      Interest under the amended credit facility accrues at the

      Lender&amp;#8217;s prime rate plus a margin of between 1.25% and

      1.75% per annum (presently 1.50% per annum), or at our option

      we can fix the rate for a period of time at LIBOR plus a

      margin of between 2.25% and 2.75% per annum (presently 2.50%

      per annum), in each case based on average credit facility

      utilization. Under the credit facility prior to the current

      amendment, the margins depended on EBITDA on a trailing

      12-month basis. Interest accrued under the credit facility

      prior to the current amendment at a rate of 5.00% per annum

      at June 30, 2013.&amp;#160;In addition, there is an unused

      commitment fee of 0.25% per year, based on a weighted average

      formula. The credit facility&amp;#8217;s expiration has been

      extended to August 2018. Our obligation to the Lender is

      presently secured by a first priority lien on substantially

      all of our non-real estate assets, and we are subject to,

      among others, a covenant that we maintain a Fixed Charge

      Coverage Ratio measured monthly on a trailing 12-month basis

      of 1.10 to 1.00 (previously 1.25 to 1.00). The covenant would

      apply only if our availability is equal to or less than the

      greater of (x) $5.0 million and (y)&amp;#160;10% of the amount of

      the credit facility (presently $75.0 million) or the

      borrowing base, whichever is less. In the event of a

      significant decrease in availability under our credit

      facility, it is highly likely that the covenant would be

      violated.&lt;/font&gt;

    &lt;/p&gt;&lt;br/&gt;</NonNumbericText><FootnoteIndexer /><CurrencyCode /><CurrencySymbol /><IsIndependantCurrency>false</IsIndependantCurrency><ShowCurrencySymbol>false</ShowCurrencySymbol><DisplayDateInUSFormat>false</DisplayDateInUSFormat></Cell></Cells><ElementDataType>nonnum:textBlockItemType</ElementDataType><SimpleDataType>na</SimpleDataType><ElementDefenition>The entire disclosure for information about short-term and long-term debt arrangements, which includes amounts of borrowings under each line of credit, note payable, commercial paper issue, bonds indenture, debenture issue, own-share lending arrangements and any other contractual agreement to repay funds, and about the underlying arrangements, rationale for a classification as long-term, including repayment terms, interest rates, collateral provided, restrictions on use of assets and activities, whether or not in compliance with debt covenants, and other matters important to users of the financial statements, such as the effects of refinancing and noncompliance with debt covenants.</ElementDefenition><ElementReferences>Reference 1: http://www.xbrl.org/2003/role/presentationRef

 -Publisher FASB

 -Name Accounting Standards Codification

 -Topic 505

 -SubTopic 10

 -Section 50

 -Paragraph 3

 -URI http://asc.fasb.org/extlink&amp;oid=6928386&amp;loc=d3e21475-112644



Reference 2: http://www.xbrl.org/2003/role/presentationRef

 -Publisher SEC

 -Name Regulation S-X (SX)

 -Number 210

 -Section 02

 -Paragraph 19, 20, 22

 -Article 5



Reference 3: http://www.xbrl.org/2003/role/presentationRef

 -Publisher FASB

 -Name Accounting Standards Codification

 -Topic 210

 -SubTopic 10

 -Section S99

 -Paragraph 1

 -Subparagraph (SX 210.5-02.19,20,22)

 -URI http://asc.fasb.org/extlink&amp;oid=6877327&amp;loc=d3e13212-122682



</ElementReferences><IsTotalLabel>false</IsTotalLabel><UnitID>0</UnitID><Label>Debt Disclosure [Text Block]</Label></Row></Rows><Footnotes /><IsEquityReport>false</IsEquityReport><ReportName>Note 4 - Loan Payable to Bank</ReportName><MonetaryRoundingLevel>UnKnown</MonetaryRoundingLevel><SharesRoundingLevel>UnKnown</SharesRoundingLevel><PerShareRoundingLevel>UnKnown</PerShareRoundingLevel><ExchangeRateRoundingLevel>UnKnown</ExchangeRateRoundingLevel><HasCustomUnits>true</HasCustomUnits><IsEmbedReport>false</IsEmbedReport><IsMultiCurrency>false</IsMultiCurrency><ReportType>Sheet</ReportType><RoleURI>http://sportchalet.com/role/Note4LoanPayabletoBank</RoleURI><NumberOfCols>1</NumberOfCols><NumberOfRows>2</NumberOfRows></InstanceReport>
