Loans and Allowance for Loan Losses, Risk Category of Loans (Details) - USD ($) $ in Thousands		Sep. 30, 2016	Dec. 31, 2015
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		$ 3,387,631	$ 3,293,304
Impaired loans included in classified loans		2,400	3,000
Commercial [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		189,795	203,415
Commercial [Member] | Pass [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		177,406	187,051
Commercial [Member] | Classified [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		12,389	16,364
Commercial [Member] | Commercial Real Estate [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		167,252	175,873
Commercial [Member] | Commercial Real Estate [Member] | Pass [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		156,034	160,243
Commercial [Member] | Commercial Real Estate [Member] | Classified [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		11,218	15,630
Commercial [Member] | Other [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		22,543	27,542
Commercial [Member] | Other [Member] | Pass [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		21,372	26,808
Commercial [Member] | Other [Member] | Classified [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		1,171	734
New York and Other States [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	[1]	2,675,893	2,650,787
New York and Other States [Member] | Commercial [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		177,290	188,414
New York and Other States [Member] | Commercial [Member] | Pass [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		164,901	172,050
New York and Other States [Member] | Commercial [Member] | Classified [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		12,389	16,364
New York and Other States [Member] | Commercial [Member] | Commercial Real Estate [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	[1]	154,747	160,965
New York and Other States [Member] | Commercial [Member] | Commercial Real Estate [Member] | Pass [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		143,529	145,335
New York and Other States [Member] | Commercial [Member] | Commercial Real Estate [Member] | Classified [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		11,218	15,630
New York and Other States [Member] | Commercial [Member] | Other [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	[1]	22,543	27,449
New York and Other States [Member] | Commercial [Member] | Other [Member] | Pass [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		21,372	26,715
New York and Other States [Member] | Commercial [Member] | Other [Member] | Classified [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		1,171	734
Florida [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		711,738	642,517
Florida [Member] | Commercial [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		12,505	15,001
Florida [Member] | Commercial [Member] | Pass [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		12,505	15,001
Florida [Member] | Commercial [Member] | Classified [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		0	0
Florida [Member] | Commercial [Member] | Commercial Real Estate [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		12,505	14,908
Florida [Member] | Commercial [Member] | Commercial Real Estate [Member] | Pass [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		12,505	14,908
Florida [Member] | Commercial [Member] | Commercial Real Estate [Member] | Classified [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		0	0
Florida [Member] | Commercial [Member] | Other [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		0	93
Florida [Member] | Commercial [Member] | Other [Member] | Pass [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		0	93
Florida [Member] | Commercial [Member] | Other [Member] | Classified [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net		$ 0	$ 0

[1] Includes New York, New Jersey, Vermont and Massachusetts.