Loans and Allowance for Loan Losses, Allowance for Loan Losses (Details) - USD ($) $ in Thousands		12 Months Ended
		Dec. 31, 2020	Dec. 31, 2019	Dec. 31, 2018
Allowance for loan losses by portfolio segment [Roll Forward]
Balance at beginning of period		$ 44,317	$ 44,766
Loans charged off [Abstract]
Total loan charge offs		661	1,207
Recoveries of loans previously charged off [Abstract]
Total recoveries		339	599
Net loans charged off (recoveries)		322	608
Provision (recoveries) for loan losses		5,600	159	$ 1,400
Balance at end of period		49,595	44,317	44,766
New York and Other States [Member]
Loans charged off [Abstract]
Total loan charge offs	[1]	627	1,130
Recoveries of loans previously charged off [Abstract]
Total recoveries	[1]	336	563
Florida [Member]
Loans charged off [Abstract]
Total loan charge offs		34	77
Recoveries of loans previously charged off [Abstract]
Total recoveries		3	36
Commercial [Member]
Allowance for loan losses by portfolio segment [Roll Forward]
Balance at beginning of period		3,999	4,048
Loans charged off [Abstract]
Total loan charge offs		36	20
Recoveries of loans previously charged off [Abstract]
Total recoveries		10	46
Net loans charged off (recoveries)		26	(26)
Provision (recoveries) for loan losses		167	(75)
Balance at end of period		4,140	3,999	4,048
Commercial [Member] | New York and Other States [Member]
Loans charged off [Abstract]
Total loan charge offs	[1]	36	20
Recoveries of loans previously charged off [Abstract]
Total recoveries	[1]	10	46
Commercial [Member] | Florida [Member]
Loans charged off [Abstract]
Total loan charge offs		0	0
Recoveries of loans previously charged off [Abstract]
Total recoveries		0	0
Real Estate Mortgage - 1 to 4 Family [Member]
Allowance for loan losses by portfolio segment [Roll Forward]
Balance at beginning of period		39,748	39,772
Loans charged off [Abstract]
Total loan charge offs		404	974
Recoveries of loans previously charged off [Abstract]
Total recoveries		317	532
Net loans charged off (recoveries)		87	442
Provision (recoveries) for loan losses		5,289	418
Balance at end of period		44,950	39,748	39,772
Real Estate Mortgage - 1 to 4 Family [Member] | New York and Other States [Member]
Loans charged off [Abstract]
Total loan charge offs	[1]	404	945
Recoveries of loans previously charged off [Abstract]
Total recoveries	[1]	314	496
Real Estate Mortgage - 1 to 4 Family [Member] | Florida [Member]
Loans charged off [Abstract]
Total loan charge offs		0	29
Recoveries of loans previously charged off [Abstract]
Total recoveries		3	36
Installment [Member]
Allowance for loan losses by portfolio segment [Roll Forward]
Balance at beginning of period		570	946
Loans charged off [Abstract]
Total loan charge offs		221	213
Recoveries of loans previously charged off [Abstract]
Total recoveries		12	21
Net loans charged off (recoveries)		209	192
Provision (recoveries) for loan losses		144	(184)
Balance at end of period		505	570	$ 946
Installment [Member] | New York and Other States [Member]
Loans charged off [Abstract]
Total loan charge offs	[1]	187	165
Recoveries of loans previously charged off [Abstract]
Total recoveries	[1]	12	21
Installment [Member] | Florida [Member]
Loans charged off [Abstract]
Total loan charge offs		34	48
Recoveries of loans previously charged off [Abstract]
Total recoveries		$ 0	$ 0

[1] Includes New York, New Jersey, Vermont and Massachusetts.