Long-Term Debt (Details Textual) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2015	Dec. 31, 2014	Dec. 31, 2007	Dec. 31, 2005	Dec. 31, 2003
Debt Instrument [Line Items]
Long-term advances from the FHLB outstanding fixed and floating rates, minimum (percent)	1.09%
Long-term advances from the FHLB outstanding fixed and floating rates, maximum (percent)	7.93%
Weighted-average interest rates (percent)	4.06%	4.16%
Unused lines of credit	$ 1,659,779
FHLB advances	0	$ 0
Prepayment penalties	$ 0	0
Interest rate swap
Debt Instrument [Line Items]
Interest rate swap agreement period	Mar. 17, 2014
PHC Statutory Trust I
Debt Instrument [Line Items]
Fixed interest rate (percent)	5.49%
Heritage Financial Statutory Trust I
Debt Instrument [Line Items]
Statutory Trust I carrying value, premium		27
Interest rate for Heritage Financial Statutory Trust I (percent)	10.20%
First M&F Statutory Trust I
Debt Instrument [Line Items]
Statutory Trust I carrying value, discount	$ 11,093	$ 11,641
Interest rate of senior note (percent)	3.795%
Capital Bancorp Capital Trust I
Debt Instrument [Line Items]
Fixed interest rate (percent)	4.42%
Junior subordinated notes	$ 91,907
Maximum
Debt Instrument [Line Items]
Maturity period of Long-term advances, new	2030
Minimum
Debt Instrument [Line Items]
Maturity period of Long-term advances, earliest	2015
LIBOR | PHC Statutory Trust I
Debt Instrument [Line Items]
Basis spread on variable rate (percent)	2.85%				2.85%
LIBOR | PHC Statutory Trust II
Debt Instrument [Line Items]
Basis spread on variable rate (percent)				1.87%
LIBOR | First M&F Statutory Trust I
Debt Instrument [Line Items]
Basis spread on variable rate (percent)	1.33%
LIBOR | Capital Bancorp Capital Trust I
Debt Instrument [Line Items]
Basis spread on variable rate (percent)	1.50%		1.50%