Loans and Allowance for Loan Losses, Risk Category of Loans (Details) (USD $)	Mar. 31, 2014		Dec. 31, 2013
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	$ 2,941,002,000		$ 2,908,809,000
Impaired loans included in classified loans	5,400,000		8,100,000
Commercial [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	220,443,000		223,481,000
Commercial [Member] | Commercial Real Estate [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	190,323,000		191,126,000
Commercial [Member] | Other [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	30,120,000		32,355,000
New York and Other States [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	2,483,028,000	[1]	2,468,322,000	[1]
New York and Other States [Member] | Commercial [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	198,273,000	[1]	202,045,000	[1]
New York and Other States [Member] | Commercial [Member] | Pass [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	189,799,000	[1]	190,715,000	[1]
New York and Other States [Member] | Commercial [Member] | Classified [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	8,474,000	[1]	11,330,000	[1]
New York and Other States [Member] | Commercial [Member] | Commercial Real Estate [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	168,226,000	[1]	169,722,000	[1]
New York and Other States [Member] | Commercial [Member] | Commercial Real Estate [Member] | Pass [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	160,244,000	[1]	159,024,000	[1]
New York and Other States [Member] | Commercial [Member] | Commercial Real Estate [Member] | Classified [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	7,982,000	[1]	10,698,000	[1]
New York and Other States [Member] | Commercial [Member] | Other [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	30,047,000	[1]	32,323,000	[1]
New York and Other States [Member] | Commercial [Member] | Other [Member] | Pass [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	29,555,000	[1]	31,691,000	[1]
New York and Other States [Member] | Commercial [Member] | Other [Member] | Classified [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	492,000	[1]	632,000	[1]
Florida [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	457,974,000		440,487,000
Florida [Member] | Commercial [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	22,170,000		21,436,000
Florida [Member] | Commercial [Member] | Pass [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	21,653,000		20,306,000
Florida [Member] | Commercial [Member] | Classified [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	517,000		1,130,000
Florida [Member] | Commercial [Member] | Commercial Real Estate [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	22,097,000		21,404,000
Florida [Member] | Commercial [Member] | Commercial Real Estate [Member] | Pass [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	21,580,000		20,274,000
Florida [Member] | Commercial [Member] | Commercial Real Estate [Member] | Classified [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	517,000		1,130,000
Florida [Member] | Commercial [Member] | Other [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	73,000		32,000
Florida [Member] | Commercial [Member] | Other [Member] | Pass [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	73,000		32,000
Florida [Member] | Commercial [Member] | Other [Member] | Classified [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans, net	$ 0		$ 0

[1] Includes New York, New Jersey, Vermont and Massachusetts.