Securities - Summary of Portfolio of Securities Available for Sale (Details) (USD $) In Thousands, unless otherwise specified	Sep. 30, 2014		Dec. 31, 2013
Schedule of Available-for-sale Securities [Line Items]
Amortized Cost	$ 238,751		$ 280,290
Gross Unrealized Gain	8,048		8,198
Gross Unrealized Loss	3,767		7,750
Fair Value	243,032		280,738
Mortgage-Related Securities
Schedule of Available-for-sale Securities [Line Items]
Amortized Cost	88,211		96,055
Gross Unrealized Gain	2,140		2,040
Gross Unrealized Loss	1,217		1,874
Fair Value	89,134		96,221
Mortgage-Related Securities | GSE certificates
Schedule of Available-for-sale Securities [Line Items]
Amortized Cost	19,295	[1]	23,759
Gross Unrealized Gain	1,397	[1]	1,442
Gross Unrealized Loss			1
Fair Value	20,692	[1]	25,200
Mortgage-Related Securities | GSE CMOs
Schedule of Available-for-sale Securities [Line Items]
Amortized Cost	59,499	[2]	62,082
Gross Unrealized Gain	743	[2]	598
Gross Unrealized Loss	1,167	[2]	1,861
Fair Value	59,075	[2]	60,819
Mortgage-Related Securities | Private label CMOs
Schedule of Available-for-sale Securities [Line Items]
Amortized Cost	9,417		10,214
Gross Unrealized Loss	50		12
Fair Value	9,367		10,202
Other Securities
Schedule of Available-for-sale Securities [Line Items]
Amortized Cost	150,540		184,235
Gross Unrealized Gain	5,908		6,158
Gross Unrealized Loss	2,550		5,876
Fair Value	153,898		184,517
Other Securities | Municipal bonds
Schedule of Available-for-sale Securities [Line Items]
Amortized Cost	964		957
Gross Unrealized Gain	133		69
Fair Value	1,097		1,026
Other Securities | Capital trust notes
Schedule of Available-for-sale Securities [Line Items]
Amortized Cost	13,428		13,419
Gross Unrealized Gain	51		60
Gross Unrealized Loss	1,870		1,681
Fair Value	11,609		11,798
Other Securities | Preferred stock
Schedule of Available-for-sale Securities [Line Items]
Amortized Cost	118,205		118,205
Gross Unrealized Gain	5,116		1,936
Gross Unrealized Loss	635		3,902
Fair Value	122,686		116,239
Other Securities | Common stock
Schedule of Available-for-sale Securities [Line Items]
Amortized Cost	17,943		51,654
Gross Unrealized Gain	608		4,093
Gross Unrealized Loss	45		293
Fair Value	$ 18,506		$ 55,454

[1] Government-sponsored enterprise

[2] Collateralized mortgage obligations