Borrowings - Summary of Interest Expense on Borrowings (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2016	Dec. 31, 2015	Dec. 31, 2014
Debt Instrument [Line Items]
Total interest expense on borrowings	$ 96,360	$ 95,784	$ 98,707
Securities sold under agreements to repurchase
Debt Instrument [Line Items]
Total interest expense on borrowings	40,484	40,373	42,626
FHLB-NY advances | Federal Home Loan Bank of New York
Debt Instrument [Line Items]
Total interest expense on borrowings	41,007	40,790	41,911
Other borrowings, net
Debt Instrument [Line Items]
Total interest expense on borrowings	13,031	13,031	13,031
Federal funds purchased
Debt Instrument [Line Items]
Total interest expense on borrowings	$ 1,838	$ 1,590	$ 1,139