Exhibit 5.1

February 19, 2015

BofI Holding, Inc.

4350 La Jolla Village Drive, Suite 140

San Diego, California 92122

Ladies and Gentlemen:

We have acted as counsel to BofI Holding, Inc., a Delaware corporation (the “Company”), in connection with its Registration Statement on Form S-3 (the “Registration Statement”) filed pursuant to the Securities Act of 1933, as amended (the “Securities Act”). The Registration Statement, including the prospectus which forms a part of the Registration Statement (the “Prospectus”), to be supplemented from time to time by one or more prospectus supplements (each, a “Prospectus Supplement”), provides for the registration of the offer and sale of:

1. shares of common stock, par value $0.01 per share (the “Common Stock”);

2. shares of preferred stock, par value $0.01 per share (the “Preferred Stock”);

3. senior debt securities in one or more series and/or subordinated debt securities in one or more series, (together, the “Debt Securities”) which may be issued under the respective indentures (each, including any supplements to the respective indentures, an “Indenture”) filed as Exhibit 4.6 and 4.8, respectively, to the Registration Statement between the Company and a trustee to be selected by the Company (the “Trustee”);

4. warrants to purchase Common Stock (the “Warrants”), which may be issued under warrant agreements, to be dated on or about the date of the first issuance of the applicable Warrants thereunder;

5. subscription rights to purchase Common Stock or other securities (the “Subscription Rights”), which may be issued under a subscription rights agent agreement to be entered into between the Company and a bank or trust company, as subscription rights agent;

6. units consisting of any combination of Common Stock, Preferred Stock, Debt Securities, Warrants, or Subscription Rights (the “Units”); and

7. $50,000,000 of common stock, par value $0.01 per share, to be sold by selling shareholders as described in the Registration Statement named in future Prospectus Supplements (the “Resale Shares”).

The Common Stock, Preferred Stock, Debt Securities, Warrants, Subscription Rights and Units are collectively referred to herein as the “Securities.” The Securities and the Resale Shares are being registered for offering and sale from time to time pursuant to Rule 415 under the Securities Act.

In connection with this opinion, we have examined and relied upon the originals, or copies certified to our satisfaction, of such records, documents, certificates, and other instruments as in our judgment are necessary or appropriate to enable us to render the opinion expressed below. Insofar as this opinion relates to factual matters, we have assumed with your permission and without independent investigation that the statements of the Company contained in the registration statement are true and correct as to all factual matters stated therein.

In rendering this opinion, we have assumed the genuineness and authenticity of all signatures on original documents; the authenticity of all documents submitted to us as originals; the conformity to originals of all documents submitted to us as copies; the accuracy, completeness, and authenticity of certificates of public officials; and the due authorization, execution, and delivery of all documents where authorization, execution, and delivery are prerequisites to the effectiveness of such documents (other than the due authorization, execution, and delivery by the Company of an Indenture). With respect to our opinion as to the Offered Common Stock (as defined below) and Offered Preferred Stock (as defined below), we have assumed that, at the time of issuance and sale, a sufficient number of shares of Common Stock and Preferred Stock, respectively, is authorized and reserved, or available for issuance. We have also assumed that with respect to any Securities being issued upon exercise of any Offered Warrants (as defined below), upon conversion of any convertible Offered Debt Securities (as defined below), upon exercise of any Offered Subscription Rights (as defined below), or upon exercise of any Offered Units (as defined below), the applicable Offered Warrants, Offered Debt Securities, Offered Subscription Rights or Offered Units will be valid and legally binding obligations of the Company, enforceable against the Company in accordance with their terms, except as enforcement

thereof may be limited by bankruptcy, insolvency, reorganization, moratorium, or other similar laws relating to or affecting creditors’ rights generally and by general equitable principles (regardless of whether such enforceability is considered in a proceeding at law or in equity).

Our opinion is expressed solely with respect to the federal laws of the United States, the General Corporation Law of the State of Delaware and, as to the Debt Securities constituting valid and legally binding obligations of the Company, the laws of the State of New York. We are not rendering any opinion as to compliance with any federal or state antifraud law, rule, or regulation relating to securities, or to the sale or issuance thereof.

On the basis of the foregoing and in reliance thereon, and subject to the qualifications herein stated, we are of the opinion that:

1. With respect to the Common Stock offered under the Registration Statement (the “Offered Common Stock”), provided that (i) the Registration Statement and any post-effective amendment thereto have become effective under the Securities Act and the Prospectus and any Prospectus Supplement(s) required by applicable laws have been delivered as required by such laws; (ii) the issuance of the Offered Common Stock has been duly authorized by all necessary corporate action on the part of the Company; (iii) the issuance and sale of the Offered Common Stock does not violate any applicable law or regulation or the Company’s Certificate of Incorporation, as amended (the “Certificate of Incorporation”), or Bylaws, as amended (the “Bylaws”) or result in a default under or breach of any agreement or instrument binding upon the Company, and complies with any applicable requirement or restriction imposed by any court or governmental body having jurisdiction over the Company; and (iv) the Offered Common Stock has been duly delivered to the purchasers thereof against payment therefor, then the Offered Common Stock, when issued and sold as contemplated in the Registration Statement, the Prospectus, and the related Prospectus Supplement(s) and in accordance with any applicable duly authorized, executed, and delivered purchase, underwriting, or similar agreement, or upon exercise of any Offered Warrants in accordance with their terms, or upon conversion of any convertible Offered Debt Securities in accordance with their terms, or upon exercise of any Subscription Rights in accordance with their terms, or upon settlement of any Offered Units in accordance with their terms, will be duly authorized, validly issued, fully paid, and nonassessable.

2. With respect to the Preferred Stock offered under the Registration Statement (the “Offered Preferred Stock”), provided that (i) the Registration Statement and any post-effective amendment thereto have become effective under the Securities Act and the Prospectus and any Prospectus Supplement(s) required by applicable laws have been delivered as required by such laws; (ii) the issuance of the Offered Preferred Stock has been duly authorized by all necessary corporate action on the part of the Company; (iii) the issuance and sale of the Offered Preferred Stock does not violate any applicable law or regulation or the Certificate of Incorporation or Bylaws or result in a default under or breach of any agreement or instrument binding upon the Company, and complies with any applicable requirement or restriction imposed by any court or governmental body having jurisdiction over the Company; and (iv) the Offered Preferred Stock has been duly delivered to the purchasers thereof against payment therefor, then the Offered Preferred Stock, when issued and sold as contemplated in the Registration Statement, the Prospectus, and the related Prospectus Supplement(s) and in accordance with any applicable duly authorized, executed, and delivered purchase, underwriting, or similar agreement, or upon conversion of any convertible Offered Debt Securities in accordance with their terms, will be duly authorized, validly issued, fully paid, and nonassessable.

3. With respect to the Warrants offered under the Registration Statement (the “Offered Warrants”), provided that (i) the Registration Statement and any post-effective amendment thereto have become effective under the Securities Act and the Prospectus and any Prospectus Supplement(s) required by applicable laws have been delivered as required by such laws; (ii) the issuance of the Offered Warrants has been duly authorized by all necessary corporate action on the part of the Company; (iii) the issuance and sale of the Offered Warrants does not violate any applicable law or regulation or the Certificate of Incorporation or Bylaws or result in a default under or breach of any agreement or instrument binding upon the Company and complies with any applicable requirement or restriction imposed by any court or governmental body having jurisdiction over the Company; and (iv) the Offered Warrants have been duly executed and delivered by the Company and authenticated by the warrant agent therefor pursuant to the applicable warrant agreement and duly delivered to the purchasers thereof against payment therefor, then the Offered Warrants, when issued and sold as contemplated in the Registration Statement, the Prospectus, and the related Prospectus Supplement(s) and in accordance with any applicable duly authorized, executed, and delivered purchase, underwriting, or similar agreement, or upon conversion of any convertible Offered Debt Securities in accordance with their terms, will be valid and legally binding obligations of the Company, enforceable against the Company in accordance with their terms, except as enforcement thereof may be limited by bankruptcy, insolvency, reorganization, moratorium, or other similar laws relating to or affecting creditors’ rights generally and by general equitable principles (regardless of whether such enforceability is considered in a proceeding at law or in equity).

4. With respect to any series of Debt Securities issued under an Indenture and offered under the Registration Statement (the “Offered Debt Securities”), provided that (i) the Registration Statement and any post-effective amendment thereto have

become effective under the Securities Act and the Prospectus and any Prospectus Supplement(s) required by applicable laws have been delivered as required by such laws; (ii) the issuance of the Offered Debt Securities and the execution and delivery of the Indenture have been duly authorized by all necessary corporate action on the part of the Company; (iii) the issuance and sale of the Offered Debt Securities and the execution and delivery of the Indenture do not violate any applicable law or regulation or the Certificate of Incorporation or Bylaws or result in a default under or breach of any agreement or instrument binding upon the Company and comply with any applicable requirement or restriction imposed by any court or governmental body having jurisdiction over the Company; and (iv) the Offered Debt Securities have been duly executed and delivered by the Company and authenticated by the Trustee pursuant to an Indenture and duly delivered to the purchasers thereof against payment therefor, then the Offered Debt Securities, when issued and sold as contemplated in the Registration Statement, the Prospectus, and the related Prospectus Supplement(s) and in accordance with any applicable duly authorized, executed and delivered purchase, underwriting or similar agreement, will be valid and legally binding obligations of the Company, enforceable against the Company in accordance with their terms, except as enforcement thereof may be limited by bankruptcy, insolvency, reorganization, moratorium, or other similar laws relating to or affecting creditors’ rights generally and by general equitable principles (regardless of whether such enforceability is considered in a proceeding at law or in equity).

5. With respect to the Subscription Rights offered under the Registration Statement (the “Offered Subscription Rights”), provided that (i) the Registration Statement and any post-effective amendment thereto have become effective under the Securities Act and the Prospectus and any Prospectus Supplement(s) required by applicable laws have been delivered as required by such laws; (ii) the issuance of the Offered Subscription Rights has been duly authorized by all necessary corporate action on the part of the Company; (iii) the issuance and sale of the Offered Subscription Rights does not violate any applicable law or regulation or the Certificate of Incorporation or Bylaws or result in a default under or breach of any agreement or instrument binding upon the Company and complies with any applicable requirement or restriction imposed by any court or governmental body having jurisdiction over the Company; and (iv) the Offered Subscription Rights have been duly executed and delivered by the Company and authenticated by the warrant agent therefor pursuant to the applicable warrant agreement and duly delivered to the purchasers thereof against payment therefor, then the Offered Subscription Rights, when issued and sold as contemplated in the Registration Statement, the Prospectus, and the related Prospectus Supplement(s) and in accordance with any applicable duly authorized, executed, and delivered purchase, underwriting, or similar agreement, or upon conversion of any convertible Offered Debt Securities in accordance with their terms will be valid and legally binding obligations of the Company, enforceable against the Company in accordance with their terms, except as enforcement thereof may be limited by bankruptcy, insolvency, reorganization, moratorium, or other similar laws relating to or affecting creditors’ rights generally and by general equitable principles (regardless of whether such enforceability is considered in a proceeding at law or in equity).

6. With respect to the Units offered under the Registration Statement (the “Offered Units”), provided that (i) the Registration Statement and any post-effective amendment thereto have become effective under the Securities Act and the Prospectus and any Prospectus Supplement(s) required by applicable laws have been delivered as required by such laws; (ii) the issuance of the Offered Units has been duly authorized by all necessary corporate action on the part of the Company; (iii) the issuance and sale of the Offered Units does not violate any applicable law or regulation or the Certificate of Incorporation or Bylaws or result in a default under or breach of any agreement or instrument binding upon the Company and complies with any applicable requirement or restriction imposed by any court or governmental body having jurisdiction over the Company; and (iv) the Offered Units have been duly executed and delivered by the Company to the purchasers thereof against payment therefor, then the Offered Units, when issued and sold as contemplated in the Registration Statement, the Prospectus and the related Prospectus Supplement(s) and in accordance with any applicable duly authorized, executed, and delivered purchase, underwriting, or similar agreement, will be valid and legally binding obligations of the Company, enforceable against the Company in accordance with their terms, except as enforcement thereof may be limited by bankruptcy, insolvency, reorganization, moratorium, or other similar laws relating to or affecting creditors’ rights generally and by general equitable principles (regardless of whether such enforceability is considered in a proceeding at law or in equity).

7. With respect to the Resale Shares offered under the Registration Statement, such Resale Shares have been duly authorized, validly issued, and are fully paid and nonassessable.

This opinion is to be used only in connection with the offer and sale of the Securities and Resale Shares while the Registration Statement is in effect and may not be used, quoted or relied upon for any other purpose.

We hereby consent to the filing of this opinion with the Commission as an exhibit to the Registration Statement in accordance with the requirements of Item 601(b)(5) of Regulation S-K under the Securities Act and to the use of this firm’s name therein and in the related prospectus under the caption “Legal Matters.” In giving such consent, we do not hereby admit that we are in the category of persons whose consent is required under Section 7 of the Securities Act or the rules and regulations of the Commission.

Very truly yours,

/s/ Loeb & Loeb LLP