Exhibit 23.1

CONSENT OF INDEPENDENT REGISTERED PUBLIC ACCOUNTING FIRM

We consent to the incorporation by reference in the Registration Statements No. 333-168839 and No. 333-170579 on Form S-8 of Territorial Bancorp Inc. of our reports dated March 15, 2018, with respect to the consolidated financial statements of Territorial Bancorp Inc. and Subsidiaries and the effectiveness of internal control over financial reporting, appearing in this Annual Report on Form 10-K for the year ended December 31, 2017.

/s/ Moss Adams LLP

Portland, Oregon

March 15, 2018