Exhibit:5.1

July 15, 2022

Impac Mortgage Holdings, Inc.

19500 Jamboree Road

Irvine, California 92612

Re:Registration Statement on Form S-4

Ladies and Gentlemen:

We have served as Maryland counsel to Impac Mortgage Holdings, Inc., a Maryland corporation (the “Company”), in connection with certain matters of Maryland law arising out of the registration of the offering and issuance of up to (a) 15,100,000 shares (the “Common Shares”) of the Company’s common stock, par value $0.01 per share (the “Common Stock”), (b) 35,000,000 shares (the “Preferred Shares”) of the Company’s 8.25% Series D Cumulative Redeemable Preferred Stock, par value $0.01 per share (the “New Preferred Stock”), and (c) 2,107,629 warrants (the “Warrants” and, together with the Common Shares and the Preferred Shares, the “Securities”) to purchase shares of Common Stock. The Securities will be issued pursuant to the Company’s offer (such offer, as it may from time to time be amended, supplemented or extended, on the terms and subject to the conditions set forth in the Registration Statement (as defined below), the “Exchange Offer”) to exchange: (1) any and all of the outstanding shares of the Company’s 9.375% Series B Cumulative Redeemable Preferred Stock, par value $0.01 per share (the “Series B Preferred Stock”), for (a) $5.00 in cash (the “Series B Cash Consideration”) or, under certain circumstances, 50 shares of the New Preferred Stock and (b) 20 shares of Common Stock, per share of Series B Preferred Stock exchanged, and (2) any and all of the outstanding shares of the Company’s 9.125% Series C Cumulative Redeemable Preferred Stock, par value $0.01 per share (the “Series C Preferred Stock” and, together with the Series B Preferred Stock, the “Outstanding Preferred Shares”), for (a) $0.10 in cash (the “Series C Cash Consideration” and, together with the Series B Cash Consideration, the “Cash Consideration”) or, under certain circumstances, one share of New Preferred Stock, (b) 1.25 shares of Common Stock and (c) 1.5 Warrants, per share of Series C Preferred Stock exchanged, pursuant to the above-referenced Registration Statement and all amendments thereto, if any (the “Registration Statement”), filed by the Company with the United States Securities and Exchange Commission (the “Commission”) under the Securities Act of 1933, as amended (the “1933 Act”).

In connection with our representation of the Company, and as a basis for the opinion hereinafter set forth, we have examined originals, or copies certified or otherwise identified to our satisfaction, of the following documents (hereinafter collectively referred to as the “Documents”):

Impac Mortgage Holdings, Inc.

July 15, 2022

Page 2

1.The Registration Statement, and the related form of prospectus included therein, substantially in the form in which it was transmitted to the Commission under the 1933 Act;

2.The charter of the Company (the “Charter”), certified by the State Department of Assessments and Taxation of Maryland (the “SDAT”);

3.The form of Articles of Amendment relating to the Series B Preferred Stock, in the form attached as Annex A-1 to the Registration Statement (the “Series B Amendment”), certified as of the date hereof by an officer of the Company;

4.The form of Articles of Amendment relating to the Series C Preferred Stock, in the form attached as Annex A-2 to the Registration Statement (the “Series C Amendment”), certified as of the date hereof by an officer of the Company;

5.The form of Articles Supplementary relating to the New Preferred Stock, in the form attached as Annex A-3 to the Registration Statement (the “Articles Supplementary”), certified as of the date hereof by an officer of the Company;

6.The form of Warrant Agent Agreement, to be entered into by and between the Company and American Stock Transfer & Trust Company, in the form attached as Annex A-1 to the Registration Statement, including the form of Warrant Certificate attached thereto evidencing any Warrants (the “Warrant Agreement”), certified as of the date hereof by an officer of the Company;

7.The Amended and Restated Bylaws of the Company (the “Bylaws”), as amended, certified as of the date hereof by an officer of the Company;

8.Resolutions (the “Resolutions”) adopted by the Board of Directors of the Company (the “Board”), relating to, among other matters, the Exchange Offer and the registration and issuance of the Securities, certified as of the date hereof by an officer of the Company;

9.A certificate of the SDAT as to the good standing of the Company, dated as of a recent date;

10.A certificate executed by an officer of the Company, dated as of the date hereof; and

11.Such other documents and matters as we have deemed necessary or appropriate to express the opinion set forth below, subject to the assumptions, limitations and qualifications stated herein.

Impac Mortgage Holdings, Inc.

July 15, 2022

Page 3

In expressing the opinion set forth below, we have assumed the following:

1.Each individual executing any of the Documents, whether on behalf of such individual or another person, is legally competent to do so.

2.Each individual executing any of the Documents on behalf of a party (other than the Company) is duly authorized to do so.

3.Each of the parties (other than the Company) executing any of the Documents has duly and validly executed and delivered each of the Documents to which such party is a signatory, and such party’s obligations set forth therein are legal, valid and binding and are enforceable in accordance with all stated terms.

4.All Documents submitted to us as originals are authentic. The form and content of all Documents submitted to us as unexecuted drafts do not differ in any respect relevant to this opinion from the form and content of such Documents as executed and delivered. All Documents submitted to us as certified or photostatic copies conform to the original documents. All signatures on all Documents are genuine. All public records reviewed or relied upon by us or on our behalf are true and complete. All representations, warranties, statements and information contained in the Documents are true and complete. There has been no oral or written modification of or amendment to any of the Documents, and there has been no waiver of any provision of any of the Documents, by action or omission of the parties or otherwise.

5.Prior to the issuance of any Securities (a) holders of at least two-thirds of the outstanding shares of Series B Preferred Stock will have validly consented to the Exchange Offer and the Series B Amendment in accordance with Maryland law, the Charter and the Bylaws, (b) holders of at least two-thirds of the outstanding shares of Series C Preferred Stock will have validly consented to the Exchange Offer and the Series C Amendment in accordance with Maryland law, the Charter and the Bylaws, (c) each of the Series B Amendment and Series C Amendment will have been approved by the holders of Common Stock in accordance with Maryland law, the Charter and the Bylaws, and (d) each of the Series B Amendment and the Series C Amendment will have been filed with, and accepted for record by, the SDAT (collectively, the “Exchange Offer Proceedings”).

6.Prior to the issuance of any Securities that are Warrants, each party to the Warrant Agreement will have duly and validly executed and delivered the Warrant Agreement, and each party’s obligations set forth in the Warrant Agreement will be legal, valid and binding and will be enforceable in accordance with the terms of the Warrant Agreement (the “Warrant Proceedings”).

Impac Mortgage Holdings, Inc.

July 15, 2022

Page 4

7.Prior to the issuance of any Securities that are Preferred Shares, the Board or a duly authorized officer of the Company will have determined that the Company is prohibited from paying the Cash Consideration in accordance with the Exchange Offer, the Registration Statement, Maryland law, the Charter, the Bylaws, and the Resolutions and the Articles Supplementary will have been filed with, and accepted for record by, the SDAT (the “Preferred Share Proceedings”).

Based upon the foregoing, and subject to the assumptions, limitations and qualifications stated herein, it is our opinion that:

1.The Company is a corporation, duly incorporated and existing under and by virtue of the laws of the State of Maryland and is in good standing with the SDAT.

2.The issuance of the Common Shares pursuant to the Exchange Offer has been duly authorized and, upon the completion of all Exchange Offer Proceedings, when and if issued and delivered upon exchange of Outstanding Preferred Shares in accordance with the terms of the Exchange Offer, the Registration Statement, the Resolutions and any other resolutions adopted by the Board, the Common Shares will be validly issued, fully paid and nonassessable.

3.The issuance of the Preferred Shares pursuant to the Exchange Offer has been duly authorized and, upon the completion of all Exchange Offer Proceedings and Preferred Share Proceedings, when and if issued and delivered upon exchange of Outstanding Preferred Shares in accordance with the terms of the Exchange Offer, the Registration Statement, the Resolutions and any other resolutions adopted by the Board, the Preferred Shares will be validly issued, fully paid and nonassessable.

4.The Company has all necessary corporate power to enter into, and perform its obligations under, the Warrant Agreement, and the execution and delivery by the Company of the Warrant Agreement, and the performance by the Company of its obligations thereunder, have been duly authorized by all necessary corporate action on the part of the Company.

5.The issuance of the Warrants pursuant to the Exchange Offer has been duly authorized and, upon the completion of all Exchange Offer Proceedings and Warrant Proceedings, when and if issued and delivered upon exchange of shares of Series C Preferred Stock in accordance with the terms of the Exchange Offer, the Warrant Agreement, the

Impac Mortgage Holdings, Inc.

July 15, 2022

Page 5

Registration Statement, the Resolutions and any other resolutions adopted by the Board, the Warrants will be validly issued.

The foregoing opinion is limited to the laws of the State of Maryland and we do not express any opinion herein concerning any federal law or any other state law. We express no opinion as to the applicability or effect of federal or state securities laws, including the securities laws of the State of Maryland, or as to federal or state laws regarding fraudulent transfers. To the extent that any matter as to which our opinion is expressed herein would be governed by the laws of any jurisdiction other than the State of Maryland, we do not express any opinion on such matter. The opinion expressed herein is subject to the effect of any judicial decision which may permit the introduction of parol evidence to modify the terms or the interpretation of agreements.

The opinion expressed herein is limited to the matters specifically set forth herein and no other opinion shall be inferred beyond the matters expressly stated. We assume no obligation to supplement this opinion if any applicable law changes after the date hereof or if we become aware of any fact that might change the opinion expressed herein after the date hereof.

This opinion is being furnished to you for submission to the Commission as an exhibit to the Registration Statement. We hereby consent to the filing of this opinion as an exhibit to the Registration Statement and to the use of the name of our firm therein. In giving this consent, we do not admit that we are within the category of persons whose consent is required by Section 7 of the 1933 Act.

Very truly yours,

/s/ Venable LLP