Office Properties and Equipment, net - Schedule of Office Properties and Equipment (Details) - USD ($) $ in Thousands	Dec. 31, 2019	Dec. 31, 2018
Property, Plant and Equipment [Line Items]
Office property and equipment, gross	$ 130,795	$ 106,528
Less accumulated depreciation and amortization	57,828	54,478
Total office properties and equipment, net	72,967	52,050
Land
Property, Plant and Equipment [Line Items]
Office property and equipment, gross	11,069	7,829
Buildings
Property, Plant and Equipment [Line Items]
Office property and equipment, gross	27,340	24,018
Land and building improvements
Property, Plant and Equipment [Line Items]
Office property and equipment, gross	31,520	24,864
Leasehold improvements
Property, Plant and Equipment [Line Items]
Office property and equipment, gross	24,038	21,279
Furniture and equipment
Property, Plant and Equipment [Line Items]
Office property and equipment, gross	$ 36,828	$ 28,538