QuickLinks -- Click here to rapidly navigate through this document

Exhibit 23.3

CONSENT OF INDEPENDENT REGISTERED CERTIFIED PUBLIC ACCOUNTING FIRM

        We consent to the reference to our firm under the caption "Experts" and to the use of our report dated December 19, 2008, included in the proxy statement/prospectus of Hanover Capital Mortgage Holdings, Inc. that is included in Amendment No. 4 to the Registration Statement (Form S-4 No. 333-155091) of Hanover Capital Mortgage Holdings, Inc.

/s/ ERNST & YOUNG LLP

Tampa, Florida
February 17, 2009

QuickLinks

Exhibit 23.3

CONSENT OF INDEPENDENT REGISTERED CERTIFIED PUBLIC ACCOUNTING FIRM