Loans and Allowance for Loan Losses, Based on Impairment Review Method (Details) (USD $) In Thousands, unless otherwise specified	Mar. 31, 2014		Dec. 31, 2013		Mar. 31, 2013	Dec. 31, 2012
Allowance for loan losses [Abstract]
Individually evaluated for impairment	$ 0		$ 0
Collectively evaluated for impairment	47,035		47,714
Total ending allowance balance	47,035		47,714		47,658	47,927
Loans [Abstract]
Individually evaluated for impairment	27,911		29,340
Collectively evaluated for impairment	2,913,091		2,879,469
Total loans	2,941,002		2,908,809
Commercial [Member]
Allowance for loan losses [Abstract]
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	3,840		4,019
Total ending allowance balance	3,840		4,019		3,964	3,771
Loans [Abstract]
Individually evaluated for impairment	5,370		8,082
Collectively evaluated for impairment	215,073		215,399
Total loans	220,443		223,481
1 to 4 Family Residential Real Estate [Member]
Allowance for loan losses [Abstract]
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	43,091		43,597
Total ending allowance balance	43,091		43,597		43,595	44,069
Loans [Abstract]
Individually evaluated for impairment	22,541		21,258
Collectively evaluated for impairment	2,692,304		2,658,175
Total loans	2,714,845		2,679,433
Installment [Member]
Allowance for loan losses [Abstract]
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	104		98
Total ending allowance balance	104		98		99	87
Loans [Abstract]
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	5,714		5,895
Total loans	5,714		5,895
New York and Other States [Member]
Loans [Abstract]
Total loans	2,483,028	[1]	2,468,322	[1]
New York and Other States [Member] | Commercial [Member]
Loans [Abstract]
Total loans	198,273	[1]	202,045	[1]
New York and Other States [Member] | Installment [Member]
Loans [Abstract]
Total loans	5,114	[1]	5,292	[1]
Florida [Member]
Loans [Abstract]
Total loans	457,974		440,487
Florida [Member] | Commercial [Member]
Loans [Abstract]
Total loans	22,170		21,436
Florida [Member] | Installment [Member]
Loans [Abstract]
Total loans	$ 600		$ 603

[1] Includes New York, New Jersey, Vermont and Massachusetts.