QuickLinks -- Click here to rapidly navigate through this document

Exhibit 23.3

CONSENT OF INDEPENDENT REGISTERED CERTIFIED PUBLIC ACCOUNTING FIRM

        We consent to the reference to our firm under the caption "Experts" and to the use of our report dated November 3, 2008, included in the Proxy Statement/Prospectus of Hanover Capital Mortgage Holdings, Inc. that is included in the Registration Statement (Form S-4) of Hanover Capital Mortgage Holdings, Inc.

/s/ ERNST & YOUNG LLP

Tampa, Florida
November 3, 2008

QuickLinks

Exhibit 23.3

CONSENT OF INDEPENDENT REGISTERED CERTIFIED PUBLIC ACCOUNTING FIRM