Investment Securities (Details) - USD ($)	Sep. 30, 2016	Dec. 31, 2015
Investment securities available-for-sale
Amortized Cost	$ 517,209,000	$ 517,601,000
Gross Unrealized Gains	7,782,000	1,649,000
Gross Unrealized Losses	696,000	6,049,000
Estimated Fair Value	524,295,000	513,201,000
Investment securities held-to-maturity
Amortized Cost	77,094,000	93,757,000
Gross Unrealized Gains	1,037,000	373,000
Gross Unrealized Losses	52,000	435,000
Estimated Fair Value	78,079,000	93,695,000
Net unrealized gain (loss), available-for-sale securities	7,100,000	(4,400,000)
Gross unrealized losses, fair value, available-for-sale securities	$ 78,078,000	$ 368,120,000
Percentage of securities in unrealized loss positions, available-for-sale securities	14.90%	71.70%
Net unrealized gain (loss), held-to-maturity securities	$ 1,000,000	$ (62,000)
Gross unrealized losses, fair value, held-to-maturity securities	$ 19,100,000	$ 52,300,000
Percentage of securities unrealized loss positions, held-to-maturity securities	24.50%	55.80%
Available for sale securities pledged as collateral	$ 443,500,000	$ 486,400,000
Debt securities
Investment securities available-for-sale
Amortized Cost	516,245,000	516,645,000
Gross Unrealized Gains	7,746,000	1,628,000
Gross Unrealized Losses	696,000	6,049,000
Estimated Fair Value	523,295,000	512,224,000
GSE debentures
Investment securities available-for-sale
Amortized Cost	64,713,000	40,658,000
Gross Unrealized Gains	1,639,000	141,000
Gross Unrealized Losses	13,000	172,000
Estimated Fair Value	66,339,000	40,627,000
Investment securities held-to-maturity
Amortized Cost	14,734,000	34,915,000
Gross Unrealized Gains	3,000	9,000
Gross Unrealized Losses	32,000	105,000
Estimated Fair Value	14,705,000	34,819,000
Net unrealized gain (loss), available-for-sale securities	1,600,000	(31,000)
Gross unrealized losses, fair value, available-for-sale securities	6,126,000	19,633,000
Net unrealized gain (loss), held-to-maturity securities	(29,000)	(96,000)
Gross unrealized losses, fair value, held-to-maturity securities	11,702,000	25,837,000
GSE CMOs
Investment securities available-for-sale
Amortized Cost	172,606,000	198,000,000
Gross Unrealized Gains	683,000	45,000
Gross Unrealized Losses	484,000	4,229,000
Estimated Fair Value	172,805,000	193,816,000
GSE MBSs
Investment securities available-for-sale
Amortized Cost	229,040,000	230,213,000
Gross Unrealized Gains	4,410,000	1,098,000
Gross Unrealized Losses	27,000	1,430,000
Estimated Fair Value	233,423,000	229,881,000
Investment securities held-to-maturity
Amortized Cost	18,710,000	19,291,000
Gross Unrealized Gains	48,000	0
Gross Unrealized Losses	6,000	305,000
Estimated Fair Value	18,752,000	18,986,000
Net unrealized gain (loss), held-to-maturity securities	42,000	(300,000)
SBA commercial loan asset-backed securities
Investment securities available-for-sale
Amortized Cost	112,000	148,000
Gross Unrealized Gains	0	0
Gross Unrealized Losses	1,000	1,000
Estimated Fair Value	111,000	147,000
Investment securities held-to-maturity
Gross unrealized losses, fair value, available-for-sale securities	103,000	139,000
Corporate debt obligations
Investment securities available-for-sale
Amortized Cost	48,306,000	46,160,000
Gross Unrealized Gains	1,014,000	344,000
Gross Unrealized Losses	16,000	18,000
Estimated Fair Value	49,304,000	46,486,000
Investment securities held-to-maturity
Net unrealized gain (loss), available-for-sale securities	1,000,000	300,000
Gross unrealized losses, fair value, available-for-sale securities	2,492,000	6,181,000
Trust preferred securities
Investment securities available-for-sale
Amortized Cost	1,468,000	1,466,000
Gross Unrealized Gains	0	0
Gross Unrealized Losses	155,000	199,000
Estimated Fair Value	1,313,000	1,267,000
Investment securities held-to-maturity
Net unrealized gain (loss), available-for-sale securities	(200,000)	(200,000)
Gross unrealized losses, fair value, available-for-sale securities	1,313,000	1,267,000
Municipal obligations
Investment securities held-to-maturity
Amortized Cost	43,150,000	39,051,000
Gross Unrealized Gains	986,000	364,000
Gross Unrealized Losses	5,000	25,000
Estimated Fair Value	44,131,000	39,390,000
Net unrealized gain (loss), held-to-maturity securities	1,000,000	300,000
Gross unrealized losses, fair value, held-to-maturity securities	2,445,000	7,629,000
Foreign government obligations
Investment securities held-to-maturity
Amortized Cost	500,000	500,000
Gross Unrealized Gains	0	0
Gross Unrealized Losses	9,000	0
Estimated Fair Value	491,000	500,000
Gross unrealized losses, fair value, held-to-maturity securities	491,000
Marketable equity securities
Investment securities available-for-sale
Amortized Cost	964,000	956,000
Gross Unrealized Gains	36,000	21,000
Gross Unrealized Losses	0	0
Estimated Fair Value	1,000,000	977,000
Investment securities held-to-maturity
Net unrealized gain (loss), available-for-sale securities	$ 36,000	$ 21,000