Loans and Allowance for Loan Losses, Allowance for Loan Losses (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2017	Dec. 31, 2016	Dec. 31, 2015
Allowance for loan losses by portfolio segment [Roll Forward]
Balance at beginning of period	$ 43,890	$ 44,762	$ 46,327
Loans charged off [Abstract]
Total loan charge offs	2,511	4,710	5,915
Recoveries of loans previously charged off [Abstract]
Total recoveries	791	888	650
Net loans charged off	1,720	3,822	5,265
Provision for loan losses	2,000	2,950	3,700
Balance at end of period	44,170	43,890	44,762
New York and Other States [Member]
Loans charged off [Abstract]
Total loan charge offs	2,325	4,545	5,578
Recoveries of loans previously charged off [Abstract]
Total recoveries	718	884	638
Florida [Member]
Loans charged off [Abstract]
Total loan charge offs	186	165	337
Recoveries of loans previously charged off [Abstract]
Total recoveries	73	4	12
Commercial [Member]
Allowance for loan losses by portfolio segment [Roll Forward]
Balance at beginning of period	4,929	4,491	4,071
Loans charged off [Abstract]
Total loan charge offs	72	795	779
Recoveries of loans previously charged off [Abstract]
Total recoveries	96	207	27
Net loans charged off	(24)	588	752
Provision for loan losses	(629)	1,026	1,172
Balance at end of period	4,324	4,929	4,491
Commercial [Member] | New York and Other States [Member]
Loans charged off [Abstract]
Total loan charge offs	72	795	779
Recoveries of loans previously charged off [Abstract]
Total recoveries	96	207	20
Commercial [Member] | Florida [Member]
Loans charged off [Abstract]
Total loan charge offs	0	0	0
Recoveries of loans previously charged off [Abstract]
Total recoveries	0	0	7
Real Estate Mortgage - 1 to 4 Family [Member]
Allowance for loan losses by portfolio segment [Roll Forward]
Balance at beginning of period	38,231	39,753	42,088
Loans charged off [Abstract]
Total loan charge offs	2,220	3,573	4,951
Recoveries of loans previously charged off [Abstract]
Total recoveries	669	617	577
Net loans charged off	1,551	2,956	4,374
Provision for loan losses	2,397	1,434	2,039
Balance at end of period	39,077	38,231	39,753
Real Estate Mortgage - 1 to 4 Family [Member] | New York and Other States [Member]
Loans charged off [Abstract]
Total loan charge offs	2,053	3,447	4,631
Recoveries of loans previously charged off [Abstract]
Total recoveries	596	613	572
Real Estate Mortgage - 1 to 4 Family [Member] | Florida [Member]
Loans charged off [Abstract]
Total loan charge offs	167	126	320
Recoveries of loans previously charged off [Abstract]
Total recoveries	73	4	5
Installment [Member]
Allowance for loan losses by portfolio segment [Roll Forward]
Balance at beginning of period	730	518	168
Loans charged off [Abstract]
Total loan charge offs	219	342	185
Recoveries of loans previously charged off [Abstract]
Total recoveries	26	64	46
Net loans charged off	193	278	139
Provision for loan losses	232	490	489
Balance at end of period	769	730	518
Installment [Member] | New York and Other States [Member]
Loans charged off [Abstract]
Total loan charge offs	200	303	168
Recoveries of loans previously charged off [Abstract]
Total recoveries	26	64	46
Installment [Member] | Florida [Member]
Loans charged off [Abstract]
Total loan charge offs	19	39	17
Recoveries of loans previously charged off [Abstract]
Total recoveries	$ 0	$ 0	$ 0