EXHIBIT 23.1

Consent of Independent Registered Public Accounting Firm

The Board of Directors and Stockholders

Heritage Financial Corporation:

We consent to the incorporation by reference in the Registration Statements (No. 333-57513, No. 333-71415, No. 333-87599, No. 333-88976, No. 333-88980, No. 333-88982, No. 333-134473, No. 333-134474, No. 333-134475, No. 333-167146, No. 333-156271 and No. 333-167145) on Forms S-8/S-3 of Heritage Financial Corporation of our report dated March 6, 2013 with respect to the consolidated financial statements of Heritage Financial Corporation, and the effectiveness of internal control over financial reporting, which report appears in this Annual Report on Form 10-K of Heritage Financial Corporation.

/s/ Crowe Horwath LLP

San Francisco, California

March 6, 2013