Loans and Allowance for Loan Losses (Details) - USD ($) $ in Thousands		Mar. 31, 2017	Dec. 31, 2016	Mar. 31, 2016	Dec. 31, 2015
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		$ 3,448,936	$ 3,430,586
Less: Allowance for loan losses		44,048	43,890	$ 44,398	$ 44,762
Net loans		3,404,888	3,386,696
Real Estate Construction Loans [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		23,900	24,800
Residential Borrowers [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		14,300	16,300
Commercial Borrowers Residential Purpose [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		9,600	8,500
Commercial [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		184,451	191,194
Less: Allowance for loan losses		4,810	4,929	4,919	4,491
Commercial [Member] | Commercial Real Estate [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		162,058	163,609
Commercial [Member] | Other [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		22,393	27,585
Real Estate Mortgage - 1 to 4 Family [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		3,256,208	3,230,574
Less: Allowance for loan losses		38,581	38,231	39,017	39,753
Real Estate Mortgage - 1 to 4 Family [Member] | First Mortgages [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		2,856,397	2,824,087
Real Estate Mortgage - 1 to 4 Family [Member] | Home Equity Loans [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		73,531	71,646
Real Estate Mortgage - 1 to 4 Family [Member] | Home Equity Lines of Credit [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		326,280	334,841
Installment [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		8,277	8,818
Less: Allowance for loan losses		657	730	$ 462	$ 518
New York and Other States [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net	[1]	2,689,048	2,692,335
New York and Other States [Member] | Commercial [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		172,731	178,905
New York and Other States [Member] | Commercial [Member] | Commercial Real Estate [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net	[1]	150,683	151,366
New York and Other States [Member] | Commercial [Member] | Other [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net	[1]	22,048	27,539
New York and Other States [Member] | Real Estate Mortgage - 1 to 4 Family [Member] | First Mortgages [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net	[1]	2,167,971	2,158,904
New York and Other States [Member] | Real Estate Mortgage - 1 to 4 Family [Member] | Home Equity Loans [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net	[1]	61,913	60,892
New York and Other States [Member] | Real Estate Mortgage - 1 to 4 Family [Member] | Home Equity Lines of Credit [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net	[1]	279,986	286,586
New York and Other States [Member] | Installment [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net	[1]	6,447	7,048
Florida [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		759,888	738,251
Florida [Member] | Commercial [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		11,720	12,289
Florida [Member] | Commercial [Member] | Commercial Real Estate [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		11,375	12,243
Florida [Member] | Commercial [Member] | Other [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		345	46
Florida [Member] | Real Estate Mortgage - 1 to 4 Family [Member] | First Mortgages [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		688,426	665,183
Florida [Member] | Real Estate Mortgage - 1 to 4 Family [Member] | Home Equity Loans [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		11,618	10,754
Florida [Member] | Real Estate Mortgage - 1 to 4 Family [Member] | Home Equity Lines of Credit [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		46,294	48,255
Florida [Member] | Installment [Member]
Loans and Leases Receivable, Net Amount [Abstract]
Total loans, net		$ 1,830	$ 1,770

[1] Includes New York, New Jersey, Vermont and Massachusetts.