Loans and Allowance for Loan Losses, Allowance for Loan Losses (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2016	Sep. 30, 2015	Sep. 30, 2016	Sep. 30, 2015
Allowance for loan losses by portfolio segment [Roll Forward]
Balance at beginning of period	$ 44,064	$ 45,571	$ 44,762	$ 46,327
Loans charged off [Abstract]
Total loan charge offs	967	1,392	3,569	4,026
Recoveries of loans previously charged off [Abstract]
Total recoveries	103	170	407	448
Net loans charged off	864	1,222	3,162	3,578
Provision (credit) for loan losses	750	800	2,350	2,400
Balance at end of period	43,950	45,149	43,950	45,149
Commercial [Member]
Allowance for loan losses by portfolio segment [Roll Forward]
Balance at beginning of period	5,046	4,022	4,491	4,071
Loans charged off [Abstract]
Total loan charge offs	356	3	688	103
Recoveries of loans previously charged off [Abstract]
Total recoveries	3	3	44	21
Net loans charged off	353	0	644	82
Provision (credit) for loan losses	505	34	1,351	67
Balance at end of period	5,198	4,056	5,198	4,056
Real Estate Mortgage - 1 to 4 Family [Member]
Allowance for loan losses by portfolio segment [Roll Forward]
Balance at beginning of period	38,589	41,087	39,753	42,088
Loans charged off [Abstract]
Total loan charge offs	551	1,335	2,631	3,793
Recoveries of loans previously charged off [Abstract]
Total recoveries	80	143	317	388
Net loans charged off	471	1,192	2,314	3,405
Provision (credit) for loan losses	(51)	752	628	1,964
Balance at end of period	38,067	40,647	38,067	40,647
Installment [Member]
Allowance for loan losses by portfolio segment [Roll Forward]
Balance at beginning of period	429	462	518	168
Loans charged off [Abstract]
Total loan charge offs	60	54	250	130
Recoveries of loans previously charged off [Abstract]
Total recoveries	20	24	46	39
Net loans charged off	40	30	204	91
Provision (credit) for loan losses	296	14	371	369
Balance at end of period	685	446	685	446
New York and Other States [Member]
Loans charged off [Abstract]
Total loan charge offs	962	1,353	3,446	3,709
Recoveries of loans previously charged off [Abstract]
Total recoveries	101	165	403	439
New York and Other States [Member] | Commercial [Member]
Loans charged off [Abstract]
Total loan charge offs	356	3	688	103
Recoveries of loans previously charged off [Abstract]
Total recoveries	3	0	44	16
New York and Other States [Member] | Real Estate Mortgage - 1 to 4 Family [Member]
Loans charged off [Abstract]
Total loan charge offs	549	1,300	2,528	3,480
Recoveries of loans previously charged off [Abstract]
Total recoveries	78	141	313	384
New York and Other States [Member] | Installment [Member]
Loans charged off [Abstract]
Total loan charge offs	57	50	230	126
Recoveries of loans previously charged off [Abstract]
Total recoveries	20	24	46	39
Florida [Member]
Loans charged off [Abstract]
Total loan charge offs	5	39	123	317
Recoveries of loans previously charged off [Abstract]
Total recoveries	2	5	4	9
Florida [Member] | Commercial [Member]
Loans charged off [Abstract]
Total loan charge offs	0	0	0	0
Recoveries of loans previously charged off [Abstract]
Total recoveries	0	3	0	5
Florida [Member] | Real Estate Mortgage - 1 to 4 Family [Member]
Loans charged off [Abstract]
Total loan charge offs	2	35	103	313
Recoveries of loans previously charged off [Abstract]
Total recoveries	2	2	4	4
Florida [Member] | Installment [Member]
Loans charged off [Abstract]
Total loan charge offs	3	4	20	4
Recoveries of loans previously charged off [Abstract]
Total recoveries	$ 0	$ 0	$ 0	$ 0