Loans Receivable and Allowance for Loan Losses - Narrative (Details)	3 Months Ended	12 Months Ended
	Dec. 31, 2017 USD ($)	Dec. 31, 2019 USD ($) loan	Dec. 31, 2018 USD ($) loan property	Dec. 31, 2017 USD ($) loan	Sep. 30, 2017 USD ($) loan
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans held-for-sale		$ 0	$ 8,100,000
Proceeds from sale of loans held-for-sale	$ 0				$ 0
Gain on sale of loans held-for-sale		718,000
Loss on sale of loans held-for-sale		0
Gain (loss) on sale of loans held-for-sale			0
Gain on sale of loans held-for-investment		67,000	618,000		0
Loss on sale of loans held-for-investment		0	0		380,000
Proceeds from sales of loans receivable	0	11,671,000	32,039,000		62,407,000
Purchases and grants of loans receivable	56,095,000	89,774,000	32,251,000		20,473,000
Carrying value of servicing liability		526,300,000	462,700,000
Servicing income	298,000	1,200,000	1,100,000		1,200,000
Loans receivable		$ 6,135,857,000	$ 4,916,840,000
Threshold period, past due status of financing receivables		30 days
Threshold period, past due for nonperforming status of financing receivables		90 days	90 days
Non-accrual loans		$ 6,687,000	$ 2,789,000
Loans past due			0
Increase in interest income if non-accrual had performed in line with their original terms	61,000	509,000	126,000		295,000
Cash basis interest income on non-accrual loans	121,000	437,000	89,000		104,000
Property acquired through foreclosure		$ 0	$ 92,000
Number of real estate properties acquired through foreclosure | property			1
Number of loans in the process of foreclosure | loan		4	14
Loans in process of foreclosure		$ 522,000	$ 1,600,000
Loan threshold for individual evaluation for impairment		500,000
Specific allowance for loan losses attributable to impaired loans		1,621,000	915,000
Impaired loans for which there are no related allowance for loan losses		10,747,000	11,647,000
Investment in troubled debt restructuring	17,600,000	$ 20,000,000.0	$ 16,000,000.0	$ 17,600,000	$ 21,100,000
Number of loans in troubled debt restructuring | loan		1	7		7
Loans receivable		$ 6,197,566,000	$ 4,979,182,000
Less Than 90 Days
Accounts, Notes, Loans and Financing Receivable [Line Items]
Non-accrual loans		$ 1,500,000	0
Number of loans in non-accrual status | loan		8
90 Days or More
Accounts, Notes, Loans and Financing Receivable [Line Items]
Investment in troubled debt restructuring	425,000		$ 101,000	$ 425,000	$ 1,000,000.0
Number of loans in troubled debt restructuring | loan		0	1	2	7
30-59 Days
Accounts, Notes, Loans and Financing Receivable [Line Items]
Investment in troubled debt restructuring		$ 660,000	$ 1,000,000.0
Number of loans in troubled debt restructuring | loan		3	7
Related Parties
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchases and grants of loans receivable	0	$ 0	$ 0		$ 390,000
Loans receivable		1,100,000	1,400,000
Freddie Mac
Accounts, Notes, Loans and Financing Receivable [Line Items]
Proceeds from sales of loans receivable	0	21,600,000	0		0
Home equity loans and advances
Accounts, Notes, Loans and Financing Receivable [Line Items]
Proceeds from sale of loans held-for-sale		97,400,000	3,600,000
Proceeds from sale of loans held-for-investment		5,300,000
Commercial real estate loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Proceeds from sale of loans held-for-sale		4,300,000
Proceeds from sale of loans held-for-investment		5,500,000
Commercial business loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Proceeds from sale of loans held-for-sale		164,000
Proceeds from sale of loans held-for-investment		901,000
Multifamily loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Proceeds from sale of loans held-for-investment			32,000,000.0		62,400,000
Multifamily and commercial
Accounts, Notes, Loans and Financing Receivable [Line Items]
Purchases and grants of loans receivable	$ 56,100,000	$ 89,800,000	32,300,000		$ 20,500,000
Financial Asset Acquired with Credit Deterioration
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans receivable			$ 0