Loans and Allowance for Loan Losses, Risk Category of Loans (Details) - USD ($) $ in Thousands		Jun. 30, 2017	Dec. 31, 2016
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		$ 3,507,473	$ 3,430,586
Non-accrual loans included in classified loans		1,700	1,800
New York and Other States [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	[1]	2,722,593	2,692,335
Florida [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		784,880	738,251
Commercial [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		183,035	191,194
Commercial [Member] | New York and Other States [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		171,587	178,905
Commercial [Member] | Florida [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		11,448	12,289
Commercial [Member] | Pass [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		171,174	174,407
Commercial [Member] | Pass [Member] | New York and Other States [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		159,726	162,118
Commercial [Member] | Pass [Member] | Florida [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		11,448	12,289
Commercial [Member] | Classified [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		11,861	16,787
Commercial [Member] | Classified [Member] | New York and Other States [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		11,861	16,787
Commercial [Member] | Classified [Member] | Florida [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		0	0
Commercial [Member] | Commercial Real Estate [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		160,344	163,609
Commercial [Member] | Commercial Real Estate [Member] | New York and Other States [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	[1]	149,245	151,366
Commercial [Member] | Commercial Real Estate [Member] | Florida [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		11,099	12,243
Commercial [Member] | Commercial Real Estate [Member] | Pass [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		150,202	148,919
Commercial [Member] | Commercial Real Estate [Member] | Pass [Member] | New York and Other States [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		139,103	136,676
Commercial [Member] | Commercial Real Estate [Member] | Pass [Member] | Florida [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		11,099	12,243
Commercial [Member] | Commercial Real Estate [Member] | Classified [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		10,142	14,690
Commercial [Member] | Commercial Real Estate [Member] | Classified [Member] | New York and Other States [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		10,142	14,690
Commercial [Member] | Commercial Real Estate [Member] | Classified [Member] | Florida [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		0	0
Commercial [Member] | Other [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		22,691	27,585
Commercial [Member] | Other [Member] | New York and Other States [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	[1]	22,342	27,539
Commercial [Member] | Other [Member] | Florida [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		349	46
Commercial [Member] | Other [Member] | Pass [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		20,972	25,488
Commercial [Member] | Other [Member] | Pass [Member] | New York and Other States [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		20,623	25,442
Commercial [Member] | Other [Member] | Pass [Member] | Florida [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		349	46
Commercial [Member] | Other [Member] | Classified [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		1,719	2,097
Commercial [Member] | Other [Member] | Classified [Member] | New York and Other States [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		1,719	2,097
Commercial [Member] | Other [Member] | Classified [Member] | Florida [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		$ 0	$ 0

[1] Includes New York, New Jersey, Vermont and Massachusetts.