Loans and Allowance for Loan Losses, Allowance for Loan Losses (Details) (USD $) In Thousands, unless otherwise specified	12 Months Ended
	Dec. 31, 2013	Dec. 31, 2012	Dec. 31, 2011
Allowance for loan losses by portfolio segment [Roll Forward]
Balance at beginning of period	$ 47,927	$ 48,717	$ 41,911
Loans charged off [Abstract]
Total loan charge offs	8,838	13,479	12,558
Recoveries of loans previously charged off [Abstract]
Total recoveries	1,625	689	614
Net loans charged off	7,213	12,790	11,944
Provision for loan losses	7,000	12,000	18,750
Balance at end of period	47,714	47,927	48,717
Commercial [Member]
Allowance for loan losses by portfolio segment [Roll Forward]
Balance at beginning of period	3,771	4,140	4,228
Loans charged off [Abstract]
Total loan charge offs	1,172	2,499	1,171
Recoveries of loans previously charged off [Abstract]
Total recoveries	519	138	59
Net loans charged off	653	2,361	1,112
Provision for loan losses	901	1,992	1,024
Balance at end of period	4,019	3,771	4,140
Real Estate Mortgage - 1 to 4 Family [Member]
Allowance for loan losses by portfolio segment [Roll Forward]
Balance at beginning of period	44,069	44,479	37,447
Loans charged off [Abstract]
Total loan charge offs	7,592	10,839	11,305
Recoveries of loans previously charged off [Abstract]
Total recoveries	1,089	502	511
Net loans charged off	6,503	10,337	10,794
Provision for loan losses	6,031	9,927	17,826
Balance at end of period	43,597	44,069	44,479
Installment [Member]
Allowance for loan losses by portfolio segment [Roll Forward]
Balance at beginning of period	87	98	236
Loans charged off [Abstract]
Total loan charge offs	74	141	82
Recoveries of loans previously charged off [Abstract]
Total recoveries	17	49	44
Net loans charged off	57	92	38
Provision for loan losses	68	81	(100)
Balance at end of period	98	87	98
New York and Other States [Member]
Loans charged off [Abstract]
Total loan charge offs	7,712	8,354	4,566
Recoveries of loans previously charged off [Abstract]
Total recoveries	746	536	575
New York and Other States [Member] | Commercial [Member]
Loans charged off [Abstract]
Total loan charge offs	1,072	1,307	171
Recoveries of loans previously charged off [Abstract]
Total recoveries	14	112	55
New York and Other States [Member] | Real Estate Mortgage - 1 to 4 Family [Member]
Loans charged off [Abstract]
Total loan charge offs	6,572	6,919	4,315
Recoveries of loans previously charged off [Abstract]
Total recoveries	715	375	477
New York and Other States [Member] | Installment [Member]
Loans charged off [Abstract]
Total loan charge offs	68	128	80
Recoveries of loans previously charged off [Abstract]
Total recoveries	17	49	43
Florida [Member]
Loans charged off [Abstract]
Total loan charge offs	1,126	5,125	7,992
Recoveries of loans previously charged off [Abstract]
Total recoveries	879	153	39
Florida [Member] | Commercial [Member]
Loans charged off [Abstract]
Total loan charge offs	100	1,192	1,000
Recoveries of loans previously charged off [Abstract]
Total recoveries	505	26	4
Florida [Member] | Real Estate Mortgage - 1 to 4 Family [Member]
Loans charged off [Abstract]
Total loan charge offs	1,020	3,920	6,990
Recoveries of loans previously charged off [Abstract]
Total recoveries	374	127	34
Florida [Member] | Installment [Member]
Loans charged off [Abstract]
Total loan charge offs	6	13	2
Recoveries of loans previously charged off [Abstract]
Total recoveries	$ 0	$ 0	$ 1