Loans - Narrative (Details)	12 Months Ended
	Dec. 31, 2020 USD ($) loan	Dec. 31, 2019 USD ($) loan	Dec. 31, 2018 USD ($)
Accounts, Notes, Loans and Financing Receivable [Line Items]
Accrued interest receivable on loans	$ 16,100,000
Net deferred loan fees	(10,000,000.0)
Financing receivable, before allowance for credit loss and fee		$ 5,140,940,000
Overdraft balances included in home equity and consumer loans	650,000	789,000
Residential loans pledged for potential borrowings at the Federal Home Loan Bank of New York	$ 2,280,000,000	$ 1,290,000,000
Number of loans still accruing | loan	1	0
Financing receivable, 90 days or more past due, still accruing	$ 1,000
Financing receivable nonaccrual troubled debt restructuring	1,100,000	$ 1,600,000
Financing receivable, troubled debt restructuring	5,000,000.0	7,300,000
Financing receivable, accrual, troubled debt restructuring	$ 3,900,000	5,700,000
Financing receivable, deferred, percentage of total loans	0.002
Loans to related parties	$ 75,700,000	55,400,000
Loans to related parties, additions	2,200,000
Repayment of loan from related parties	6,600,000
Loans to related parties, commenced or ceased	24,700,000
Loans held for sale	1,335,000	1,743,000
Residential real estate acquired through disclosure	0	563,000
Other repossessed assets owned	0	0
Writedown of other repossessed assets	39,000	153,000	$ 70,000
Cumulated adjustment for adoption of ASU | Accounting Standards Update 2016-13
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, purchased with credit deterioration, allowance for credit loss at acquisition date		1,300,000
COVID-19
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, deferred	9,700,000
Paycheck Protection Program ("PPP")
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, before allowance for credit loss and fee	284,600,000
Residential mortgage
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, before allowance for credit loss and fee		335,191,000
Financing receivable, 90 days or more past due, still accruing	0
Financing receivable, nonaccrual, in the process of foreclosure	1,700,000
Consumer
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, before allowance for credit loss and fee		337,977,000
Financing receivable, 90 days or more past due, still accruing	1,000
Financing receivable, nonaccrual, in the process of foreclosure		2,000,000.0
Commercial, industrial and other
Accounts, Notes, Loans and Financing Receivable [Line Items]
Financing receivable, before allowance for credit loss and fee		$ 431,934,000
Financing receivable, 90 days or more past due, still accruing	$ 0