Exhibit 23.1

Consent of Independent Registered Public Accounting Firm

The Board of Directors

Affirmative Insurance Holdings, Inc.:

We consent to the incorporation by reference in the registration statement (No. 333-137938) on Form S-8 of Affirmative Insurance Holdings, Inc. and subsidiaries of our reports dated March 30, 2012, with respect to the consolidated balance sheets of Affirmative Insurance Holdings, Inc. and subsidiaries as of December 31, 2011 and 2010, and the related consolidated statements of income (loss), stockholders’ equity (deficit), comprehensive income (loss), and cash flows for each of the years in the three-year period ended December 31, 2011, and the effectiveness of internal control over financial reporting as of December 31, 2011, which reports appear in the December 31, 2011 annual report on Form 10-K of Affirmative Insurance Holdings, Inc. and subsidiaries.

/s/ KPMG LLP

Dallas, Texas

March 30, 2012