Loans and Allowance for Loan Losses, Based on Impairment Review Method (Details) (USD $) In Thousands, unless otherwise specified	Jun. 30, 2014		Mar. 31, 2014	Dec. 31, 2013		Jun. 30, 2013	Mar. 31, 2013	Dec. 31, 2012
Allowance for loan losses [Abstract]
Individually evaluated for impairment	$ 0			$ 0
Collectively evaluated for impairment	46,935			47,714
Total ending allowance balance	46,935		47,035	47,714		47,589	47,658	47,927
Loans [Abstract]
Individually evaluated for impairment	27,354			29,340
Collectively evaluated for impairment	2,978,796			2,879,469
Total loans	3,006,150			2,908,809
Commercial [Member]
Allowance for loan losses [Abstract]
Individually evaluated for impairment	0			0
Collectively evaluated for impairment	4,073			4,019
Total ending allowance balance	4,073		3,840	4,019		3,719	3,964	3,771
Loans [Abstract]
Individually evaluated for impairment	5,949			8,082
Collectively evaluated for impairment	216,706			215,399
Total loans	222,655			223,481
1 to 4 Family Residential Real Estate [Member]
Allowance for loan losses [Abstract]
Individually evaluated for impairment	0			0
Collectively evaluated for impairment	42,752			43,597
Total ending allowance balance	42,752		43,091	43,597		43,766	43,595	44,069
Loans [Abstract]
Individually evaluated for impairment	21,405			21,258
Collectively evaluated for impairment	2,755,992			2,658,175
Total loans	2,777,397			2,679,433
Installment [Member]
Allowance for loan losses [Abstract]
Individually evaluated for impairment	0			0
Collectively evaluated for impairment	110			98
Total ending allowance balance	110		104	98		104	99	87
Loans [Abstract]
Individually evaluated for impairment	0			0
Collectively evaluated for impairment	6,098			5,895
Total loans	6,098			5,895
New York and Other States [Member]
Loans [Abstract]
Total loans	2,517,744	[1]		2,468,322	[1]
Florida [Member]
Loans [Abstract]
Total loans	$ 488,406			$ 440,487

[1] Includes New York, New Jersey, Vermont and Massachusetts.