Loans Receivable and Allowance for Loan Losses - Narrative (Details)	12 Months Ended
	Dec. 31, 2021 USD ($) loan	Dec. 31, 2020 USD ($) loan	Dec. 31, 2019 USD ($) loan
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans held-for-sale	$ 0	$ 4,100,000
Proceeds from sales of loans held-for-sale	302,039,000	111,764,000	$ 101,946,000
Gain on sale of loans held-for-sale	8,600,000	1,700,000	718,000
Loss on sale of loans held-for-sale	24,000	0	0
Proceeds from sales of loans receivable	0	35,613,000	11,671,000
Gain on sale of loans held-for-investment		161,000	67,000
Loss on sale of loans held-for-investment		0	0
Purchases and grants of loans receivable	85,382,000	0	89,774,000
Carrying value of servicing liability	519,500,000	598,000,000
Servicing income	$ 1,500,000	$ 1,400,000	1,200,000
Threshold period, past due status of financing receivables	30 days
Threshold period, past due for nonperforming status of financing receivables	90 days	90 days
Non-accrual loans	$ 3,900,000	$ 8,200,000
Increase in interest income if non-accrual had performed in line with their original terms	190,000	426,000	509,000
Cash basis interest income on non-accrual loans	242,000	410,000	$ 437,000
Loans past due	0	0
Property acquired through foreclosure	$ 0	$ 0
Number of loans in the process of foreclosure | loan	1	2
Loans in process of foreclosure	$ 87,000	$ 398,000
Loan threshold for individual evaluation for impairment	500,000
Specific allowance for loan losses attributable to impaired loans	378,000	1,088,000
Impaired loans for which there are no related allowance for loan losses	17,280,000	19,061,000
Investment in troubled debt restructuring	$ 22,400,000	$ 45,400,000
Number of loans in troubled debt restructuring | loan	3	7	1
Less Than 90 Days
Accounts, Notes, Loans and Financing Receivable [Line Items]
Non-accrual loans	$ 1,700,000	$ 2,200,000
Number of loans in non-accrual status | loan	10	19
90 Days or More
Accounts, Notes, Loans and Financing Receivable [Line Items]
Investment in troubled debt restructuring		$ 91,000
Number of loans in troubled debt restructuring | loan	0	1
30-59 Days
Accounts, Notes, Loans and Financing Receivable [Line Items]
Investment in troubled debt restructuring	$ 36,000	$ 11,900,000
Number of loans in troubled debt restructuring | loan	1	3
Related Parties
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchases and grants of loans receivable	$ 522,700	$ 300,000	$ 0
Number of loans granted to related party | loan	1	1
Freddie Mac
Accounts, Notes, Loans and Financing Receivable [Line Items]
Proceeds from sales of loans receivable	$ 99,600,000	$ 117,300,000	21,600,000
Gain on sale of loans receivable	2,300,000	3,500,000
Loss on sale of loans receivable	0	0
Real estate loans | One-to-four family
Accounts, Notes, Loans and Financing Receivable [Line Items]
Proceeds from sales of loans held-for-sale	18,500,000	111,800,000	97,400,000
Proceeds from sale of loans held-for-investment		15,100,000	5,300,000
Purchases and grants of loans receivable	11,800,000		89,800,000
Specific allowance for loan losses attributable to impaired loans	258,000	391,000
Impaired loans for which there are no related allowance for loan losses	$ 1,882,000	$ 3,344,000
Number of loans in troubled debt restructuring | loan	2	0
Real estate loans | Multifamily and commercial
Accounts, Notes, Loans and Financing Receivable [Line Items]
Proceeds from sales of loans held-for-sale	$ 19,100,000		4,300,000
Proceeds from sale of loans held-for-investment			901,000
Purchases and grants of loans receivable	73,600,000
Specific allowance for loan losses attributable to impaired loans	97,000	$ 601,000
Impaired loans for which there are no related allowance for loan losses	$ 14,623,000	$ 13,058,000
Number of loans in troubled debt restructuring | loan	1	5
Real estate loans | Construction
Accounts, Notes, Loans and Financing Receivable [Line Items]
Proceeds from sales of loans held-for-sale	$ 6,400,000
Proceeds from sale of loans held-for-investment		$ 13,000,000
Commercial business loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Proceeds from sales of loans held-for-sale	258,100,000		$ 164,000
Proceeds from sale of loans held-for-investment		7,600,000
Specific allowance for loan losses attributable to impaired loans	16,000	84,000
Impaired loans for which there are no related allowance for loan losses	$ 573,000	$ 1,945,000
Number of loans in troubled debt restructuring | loan	0	2	1
Commercial business loans | Paycheck Protection Program
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, unamortized loan fee (cost)	$ 1,200,000	$ 6,600,000
Consumer loans | Home equity loans and advances
Accounts, Notes, Loans and Financing Receivable [Line Items]
Proceeds from sale of loans held-for-investment			$ 5,500,000
Specific allowance for loan losses attributable to impaired loans	7,000	12,000
Impaired loans for which there are no related allowance for loan losses	$ 202,000	$ 714,000