Loans and Allowance for Loan Losses, Risk Category of Loans (Details) - USD ($) $ in Thousands		Mar. 31, 2017	Dec. 31, 2016
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		$ 3,448,936	$ 3,430,586
Impaired loans included in classified loans		1,800	1,800
Commercial [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		184,451	191,194
Commercial [Member] | Pass [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		169,117	174,407
Commercial [Member] | Classified [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		15,334	16,787
Commercial [Member] | Commercial Real Estate [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		162,058	163,609
Commercial [Member] | Commercial Real Estate [Member] | Pass [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		148,857	148,919
Commercial [Member] | Commercial Real Estate [Member] | Classified [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		13,201	14,690
Commercial [Member] | Other [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		22,393	27,585
Commercial [Member] | Other [Member] | Pass [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		20,260	25,488
Commercial [Member] | Other [Member] | Classified [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		2,133	2,097
New York and Other States [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	[1]	2,689,048	2,692,335
New York and Other States [Member] | Commercial [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		172,731	178,905
New York and Other States [Member] | Commercial [Member] | Pass [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		157,397	162,118
New York and Other States [Member] | Commercial [Member] | Classified [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		15,334	16,787
New York and Other States [Member] | Commercial [Member] | Commercial Real Estate [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	[1]	150,683	151,366
New York and Other States [Member] | Commercial [Member] | Commercial Real Estate [Member] | Pass [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		137,482	136,676
New York and Other States [Member] | Commercial [Member] | Commercial Real Estate [Member] | Classified [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		13,201	14,690
New York and Other States [Member] | Commercial [Member] | Other [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	[1]	22,048	27,539
New York and Other States [Member] | Commercial [Member] | Other [Member] | Pass [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		19,915	25,442
New York and Other States [Member] | Commercial [Member] | Other [Member] | Classified [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		2,133	2,097
Florida [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		759,888	738,251
Florida [Member] | Commercial [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		11,720	12,289
Florida [Member] | Commercial [Member] | Pass [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		11,720	12,289
Florida [Member] | Commercial [Member] | Classified [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		0	0
Florida [Member] | Commercial [Member] | Commercial Real Estate [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		11,375	12,243
Florida [Member] | Commercial [Member] | Commercial Real Estate [Member] | Pass [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		11,375	12,243
Florida [Member] | Commercial [Member] | Commercial Real Estate [Member] | Classified [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		0	0
Florida [Member] | Commercial [Member] | Other [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		345	46
Florida [Member] | Commercial [Member] | Other [Member] | Pass [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		345	46
Florida [Member] | Commercial [Member] | Other [Member] | Classified [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		$ 0	$ 0

[1] Includes New York, New Jersey, Vermont and Massachusetts.