Loans and Allowance for Loan Losses (Details) - USD ($) $ in Thousands		Jun. 30, 2016	Mar. 31, 2016	Dec. 31, 2015	Jun. 30, 2015	Mar. 31, 2015	Dec. 31, 2014
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		$ 3,343,194		$ 3,293,304
Less: Allowance for loan losses		44,064	$ 44,398	44,762	$ 45,571	$ 45,944	$ 46,327
Net loans		3,299,130		3,248,542
Real Estate Construction Loans [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		22,800		26,600
Residential Borrowers [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		13,400		16,000
Commercial Borrowers Residential Purpose [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		9,400		10,600
Commercial [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		195,698		203,415
Less: Allowance for loan losses		5,046	4,919	4,491	4,022	4,024	4,071
Commercial [Member] | Commercial Real Estate [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		172,422		175,873
Commercial [Member] | Other [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		23,276		27,542
Real Estate Mortgage - 1 to 4 Family [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		3,139,020		3,080,498
Less: Allowance for loan losses		38,589	39,017	39,753	41,087	41,529	42,088
Real Estate Mortgage - 1 to 4 Family [Member] | First Mortgages [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		2,720,118		2,660,672
Real Estate Mortgage - 1 to 4 Family [Member] | Home Equity Loans [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		66,833		60,501
Real Estate Mortgage - 1 to 4 Family [Member] | Home Equity Lines of Credit [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		352,069		359,325
Installment [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		8,476		9,391
Less: Allowance for loan losses		429	$ 462	518	$ 462	$ 391	$ 168
New York and Other States [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net	[1]	2,662,277		2,650,787
New York and Other States [Member] | Commercial [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		181,346		188,414
New York and Other States [Member] | Commercial [Member] | Commercial Real Estate [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net	[1]	158,150		160,965
New York and Other States [Member] | Commercial [Member] | Other [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net	[1]	23,196		27,449
New York and Other States [Member] | Real Estate Mortgage - 1 to 4 Family [Member] | First Mortgages [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net	[1]	2,116,506		2,093,957
New York and Other States [Member] | Real Estate Mortgage - 1 to 4 Family [Member] | Home Equity Loans [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net	[1]	56,580		52,251
New York and Other States [Member] | Real Estate Mortgage - 1 to 4 Family [Member] | Home Equity Lines of Credit [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net	[1]	300,952		308,165
New York and Other States [Member] | Installment [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net	[1]	6,893		8,000
Florida [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		680,917		642,517
Florida [Member] | Commercial [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		14,352		15,001
Florida [Member] | Commercial [Member] | Commercial Real Estate [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		14,272		14,908
Florida [Member] | Commercial [Member] | Other [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		80		93
Florida [Member] | Real Estate Mortgage - 1 to 4 Family [Member] | First Mortgages [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		603,612		566,715
Florida [Member] | Real Estate Mortgage - 1 to 4 Family [Member] | Home Equity Loans [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		10,253		8,250
Florida [Member] | Real Estate Mortgage - 1 to 4 Family [Member] | Home Equity Lines of Credit [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		51,117		51,160
Florida [Member] | Installment [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		$ 1,583		$ 1,391

[1] Includes New York, New Jersey, Vermont and Massachusetts.