Loans and Allowance for Loan Losses, Allowance for Loan Losses (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2017	Mar. 31, 2016
Allowance for loan losses by portfolio segment [Roll Forward]
Balance at beginning of period	$ 43,890	$ 44,762
Loans charged off [Abstract]
Total loan charge offs	629	1,334
Recoveries of loans previously charged off [Abstract]
Total recoveries	187	170
Net loans charged off	442	1,164
Provision for loan losses	600	800
Balance at end of period	44,048	44,398
Commercial [Member]
Allowance for loan losses by portfolio segment [Roll Forward]
Balance at beginning of period	4,929	4,491
Loans charged off [Abstract]
Total loan charge offs	72	264
Recoveries of loans previously charged off [Abstract]
Total recoveries	8	40
Net loans charged off	64	224
Provision for loan losses	(55)	652
Balance at end of period	4,810	4,919
Real Estate Mortgage - 1 to 4 Family [Member]
Allowance for loan losses by portfolio segment [Roll Forward]
Balance at beginning of period	38,231	39,753
Loans charged off [Abstract]
Total loan charge offs	514	973
Recoveries of loans previously charged off [Abstract]
Total recoveries	169	119
Net loans charged off	345	854
Provision for loan losses	695	118
Balance at end of period	38,581	39,017
Installment [Member]
Allowance for loan losses by portfolio segment [Roll Forward]
Balance at beginning of period	730	518
Loans charged off [Abstract]
Total loan charge offs	43	97
Recoveries of loans previously charged off [Abstract]
Total recoveries	10	11
Net loans charged off	33	86
Provision for loan losses	(40)	30
Balance at end of period	657	462
New York and Other States [Member]
Loans charged off [Abstract]
Total loan charge offs	543	1,234
Recoveries of loans previously charged off [Abstract]
Total recoveries	187	169
New York and Other States [Member] | Commercial [Member]
Loans charged off [Abstract]
Total loan charge offs	72	264
Recoveries of loans previously charged off [Abstract]
Total recoveries	8	40
New York and Other States [Member] | Real Estate Mortgage - 1 to 4 Family [Member]
Loans charged off [Abstract]
Total loan charge offs	430	889
Recoveries of loans previously charged off [Abstract]
Total recoveries	169	118
New York and Other States [Member] | Installment [Member]
Loans charged off [Abstract]
Total loan charge offs	41	81
Recoveries of loans previously charged off [Abstract]
Total recoveries	10	11
Florida [Member]
Loans charged off [Abstract]
Total loan charge offs	86	100
Recoveries of loans previously charged off [Abstract]
Total recoveries	0	1
Florida [Member] | Commercial [Member]
Loans charged off [Abstract]
Total loan charge offs	0	0
Recoveries of loans previously charged off [Abstract]
Total recoveries	0	0
Florida [Member] | Real Estate Mortgage - 1 to 4 Family [Member]
Loans charged off [Abstract]
Total loan charge offs	84	84
Recoveries of loans previously charged off [Abstract]
Total recoveries	0	1
Florida [Member] | Installment [Member]
Loans charged off [Abstract]
Total loan charge offs	2	16
Recoveries of loans previously charged off [Abstract]
Total recoveries	$ 0	$ 0