Allowance for Non-Covered Loan Losses and Credit Quality (Details) (USD $)	12 Months Ended
	Dec. 31, 2013 contract	Dec. 31, 2012 contract	Dec. 31, 2011
Accounts, Notes, Loans and Financing Receivable [Line Items]
Minimum period within which non covered nonaccrual restructured loans and non covered past due	90 days
Financing Receivable Additional Allowance For Credit Losses for Covered Loans	$ 14,600,000	$ 3,300,000	$ 870,000
Loans, Allowance for Credit Losses [Roll Forward]
Balance at beginning of period	17,147,000	18,032,000
Provision	1,875,000	7,100,000
Charge-offs	(3,296,000)	(9,183,000)
Recoveries	1,367,000	1,198,000
Balance at end of period	17,093,000	17,147,000	18,032,000
Loans, Allowance for Credit Loss, Additional Information [Abstract]
Individually evaluated for impairment	4,406,000	3,555,000
Collectively evaluated for impairment	12,687,000	13,592,000
Total allowance for loan losses	17,093,000	17,147,000
Non-covered loans
Individually evaluated for impairment	32,818,000	41,854,000
Collectively evaluated for impairment	854,165,000	809,542,000
Total non-covered loans	886,983,000	851,396,000
Deferred loan costs, net	1,703,000	1,738,000
Impaired Loans Receivable, Recorded Investment [Abstract]
With no related allowance recorded	12,241,000	24,114,000
With an allowance recorded	20,577,000	17,740,000
Related Allowance	4,406,000	3,555,000
Total	32,818,000	41,854,000
Impaired Loans, Unpaid Principal Balance [Abstract]
Loans with no related allowance, unpaid principal balance	16,688,000	33,796,000
Loans with related allowance, unpaid principal balance	20,979,000	18,125,000
Total impaired non-covered loans, unpaid principal balance, Total	37,667,000	51,921,000
Impaired Loans, Average Recorded Investment [Abstract]
Average recorded investment in non-covered impaired loans	37,300,000	45,200,000	47,200,000
Impaired Financing Receivable, Interest Income, Accrual Method [Abstract]
Interest income on non-covered impaired loans	438,000	605,000	375,000
Impaired loans, interest income forgone	151,000	600,000	1,700,000
Troubled Debt Restructurings by Status [Abstract]
Accrual status	24,409,000	26,303,000
Nonaccrual status	6,365,000	12,356,000
Total modifications	30,774,000	38,659,000
Troubled Debt Restructuring (TDR) [Abstract]
Number of loans restructured (in contracts)	13	26
Pre-modification investment	3,784,000	7,404,000
Post-modification investment	3,690,000	6,853,000
Non-covered nonaccrual loans and past due loans, by class
30 - 59 Days past due	2,810,000	1,667,000
60 - 89 Days past due	368,000	963,000
Greater than 90 days and accruing	0	0
Total past due	3,178,000	2,630,000
Nonaccrual	8,409,000	15,551,000
Current	875,396,000	833,215,000
Total non-covered loans	886,983,000	851,396,000
Deferred loan costs, net	1,703,000	1,738,000
Total non-covered loans, gross	888,686,000	853,134,000
Pass/Watch [Member]
Non-covered loans
Total non-covered loans	780,242,000	740,535,000
Special Mention [Member]
Non-covered loans
Total non-covered loans	40,592,000	33,552,000
Substandard [Member]
Non-covered loans
Total non-covered loans	66,149,000	77,309,000
Doubtful [Member]
Non-covered loans
Total non-covered loans	0	0
Commercial [Member]
Loans, Allowance for Credit Losses [Roll Forward]
Balance at beginning of period	4,405,000	4,034,000
Provision	245,000	1,763,000
Charge-offs	(562,000)	(1,640,000)
Recoveries	235,000	248,000
Balance at end of period	4,323,000	4,405,000
Loans, Allowance for Credit Loss, Additional Information [Abstract]
Individually evaluated for impairment	1,578,000	1,468,000
Collectively evaluated for impairment	2,745,000	2,937,000
Total allowance for loan losses	4,323,000	4,405,000
Non-covered loans
Individually evaluated for impairment	8,254,000	9,974,000
Collectively evaluated for impairment	171,660,000	152,507,000
Total non-covered loans	179,914,000	162,481,000
Impaired Loans Receivable, Recorded Investment [Abstract]
With no related allowance recorded	2,558,000	3,737,000
With an allowance recorded	5,696,000	6,237,000
Related Allowance	1,578,000	1,468,000
Impaired Loans, Unpaid Principal Balance [Abstract]
Loans with no related allowance, unpaid principal balance	3,036,000	4,231,000
Loans with related allowance, unpaid principal balance	5,711,000	6,237,000
Troubled Debt Restructurings by Status [Abstract]
Accrual status	6,077,000	7,008,000
Nonaccrual status	506,000	1,160,000
Total modifications	6,583,000	8,168,000
Troubled Debt Restructuring (TDR) [Abstract]
Number of loans restructured (in contracts)	2	18
Pre-modification investment	107,000	4,812,000
Post-modification investment	106,000	4,403,000
Troubled debt restructuring that subsequently defaulted		557,000
Non-covered nonaccrual loans and past due loans, by class
30 - 59 Days past due	449,000	232,000
60 - 89 Days past due	51,000	373,000
Greater than 90 days and accruing	0	0
Total past due	500,000	605,000
Nonaccrual	2,176,000	2,966,000
Current	177,238,000	158,910,000
Total non-covered loans	179,914,000	162,481,000
Commercial [Member] | Pass/Watch [Member]
Non-covered loans
Total non-covered loans	158,571,000	140,809,000
Commercial [Member] | Special Mention [Member]
Non-covered loans
Total non-covered loans	6,340,000	4,412,000
Commercial [Member] | Substandard [Member]
Non-covered loans
Total non-covered loans	15,003,000	17,260,000
Commercial [Member] | Doubtful [Member]
Non-covered loans
Total non-covered loans	0	0
Real Estate Mortgage [Member]
Loans, Allowance for Credit Losses [Roll Forward]
Balance at beginning of period	6,516,000	6,500,000
Provision	353,000	2,889,000
Charge-offs	(225,000)	(3,311,000)
Recoveries	183,000	438,000
Balance at end of period	6,827,000	6,516,000
Loans, Allowance for Credit Loss, Additional Information [Abstract]
Individually evaluated for impairment	1,558,000	1,154,000
Collectively evaluated for impairment	5,269,000	5,362,000
Total allowance for loan losses	6,827,000	6,516,000
Non-covered loans
Individually evaluated for impairment	12,612,000	11,357,000
Collectively evaluated for impairment	476,536,000	441,270,000
Total non-covered loans	489,148,000	452,627,000
Impaired Loans Receivable, Recorded Investment [Abstract]
With no related allowance recorded	3,310,000	4,543,000
With an allowance recorded	9,302,000	6,814,000
Related Allowance	1,558,000	1,154,000
Impaired Loans, Unpaid Principal Balance [Abstract]
Loans with no related allowance, unpaid principal balance	3,610,000	5,415,000
Loans with related allowance, unpaid principal balance	9,688,000	7,181,000
Troubled Debt Restructurings by Status [Abstract]
Accrual status	11,222,000	8,566,000
Nonaccrual status	1,163,000	2,163,000
Total modifications	12,385,000	10,729,000
Troubled Debt Restructuring (TDR) [Abstract]
Number of loans restructured (in contracts)	8	5
Pre-modification investment	3,479,000	1,266,000
Post-modification investment	3,479,000	1,255,000
Non-covered nonaccrual loans and past due loans, by class
30 - 59 Days past due	939,000	0
60 - 89 Days past due	211,000	0
Greater than 90 days and accruing	0	0
Total past due	1,150,000	0
Nonaccrual	1,390,000	2,792,000
Current	486,608,000	449,835,000
Total non-covered loans	489,148,000	452,627,000
Real Estate Mortgage [Member] | Pass/Watch [Member]
Non-covered loans
Total non-covered loans	425,112,000	394,919,000
Real Estate Mortgage [Member] | Special Mention [Member]
Non-covered loans
Total non-covered loans	29,570,000	27,426,000
Real Estate Mortgage [Member] | Substandard [Member]
Non-covered loans
Total non-covered loans	34,466,000	30,282,000
Real Estate Mortgage [Member] | Doubtful [Member]
Non-covered loans
Total non-covered loans	0	0
Real Estate Mortgage [Member] | One to Four Family Residential [Member]
Non-covered loans
Total non-covered loans	41,811,000	36,873,000
Impaired Loans Receivable, Recorded Investment [Abstract]
With no related allowance recorded	544,000	938,000
With an allowance recorded	728,000	524,000
Related Allowance	81,000	53,000
Impaired Loans, Unpaid Principal Balance [Abstract]
Loans with no related allowance, unpaid principal balance	629,000	1,132,000
Loans with related allowance, unpaid principal balance	728,000	524,000
Troubled Debt Restructurings by Status [Abstract]
Accrual status	765,000	573,000
Nonaccrual status	320,000	340,000
Total modifications	1,085,000	913,000
Troubled Debt Restructuring (TDR) [Abstract]
Number of loans restructured (in contracts)	1	3
Pre-modification investment	213,000	565,000
Post-modification investment	213,000	561,000
Non-covered nonaccrual loans and past due loans, by class
30 - 59 Days past due	0	0
60 - 89 Days past due	211,000	0
Greater than 90 days and accruing	0	0
Total past due	211,000	0
Nonaccrual	507,000	889,000
Current	41,093,000	35,984,000
Total non-covered loans	41,811,000	36,873,000
Real Estate Mortgage [Member] | One to Four Family Residential [Member] | Pass/Watch [Member]
Non-covered loans
Total non-covered loans	37,720,000	31,511,000
Real Estate Mortgage [Member] | One to Four Family Residential [Member] | Special Mention [Member]
Non-covered loans
Total non-covered loans	30,000	1,139,000
Real Estate Mortgage [Member] | One to Four Family Residential [Member] | Substandard [Member]
Non-covered loans
Total non-covered loans	4,061,000	4,223,000
Real Estate Mortgage [Member] | One to Four Family Residential [Member] | Doubtful [Member]
Non-covered loans
Total non-covered loans	0	0
Real Estate Mortgage [Member] | Multi Family and Commercial [Member]
Non-covered loans
Total non-covered loans	447,337,000	415,754,000
Impaired Loans Receivable, Recorded Investment [Abstract]
With no related allowance recorded	2,766,000	3,605,000
With an allowance recorded	8,574,000	6,290,000
Related Allowance	1,477,000	1,101,000
Impaired Loans, Unpaid Principal Balance [Abstract]
Loans with no related allowance, unpaid principal balance	2,981,000	4,283,000
Loans with related allowance, unpaid principal balance	8,960,000	6,657,000
Troubled Debt Restructurings by Status [Abstract]
Accrual status	10,457,000	7,993,000
Nonaccrual status	843,000	1,823,000
Total modifications	11,300,000	9,816,000
Troubled Debt Restructuring (TDR) [Abstract]
Number of loans restructured (in contracts)	7	2
Pre-modification investment	3,266,000	701,000
Post-modification investment	3,266,000	694,000
Troubled debt restructuring that subsequently defaulted		37,000
Non-covered nonaccrual loans and past due loans, by class
30 - 59 Days past due	939,000	0
60 - 89 Days past due	0	0
Greater than 90 days and accruing	0	0
Total past due	939,000	0
Nonaccrual	883,000	1,903,000
Current	445,515,000	413,851,000
Total non-covered loans	447,337,000	415,754,000
Real Estate Mortgage [Member] | Multi Family and Commercial [Member] | Pass/Watch [Member]
Non-covered loans
Total non-covered loans	387,392,000	363,408,000
Real Estate Mortgage [Member] | Multi Family and Commercial [Member] | Special Mention [Member]
Non-covered loans
Total non-covered loans	29,540,000	26,287,000
Real Estate Mortgage [Member] | Multi Family and Commercial [Member] | Substandard [Member]
Non-covered loans
Total non-covered loans	30,405,000	26,059,000
Real Estate Mortgage [Member] | Multi Family and Commercial [Member] | Doubtful [Member]
Non-covered loans
Total non-covered loans	0	0
Real Estate Construction [Member]
Loans, Allowance for Credit Losses [Roll Forward]
Balance at beginning of period	3,050,000	4,046,000
Provision	(372,000)	1,699,000
Charge-offs	(70,000)	(2,706,000)
Recoveries	442,000	11,000
Balance at end of period	3,050,000	3,050,000
Loans, Allowance for Credit Loss, Additional Information [Abstract]
Individually evaluated for impairment	1,226,000	881,000
Collectively evaluated for impairment	1,824,000	2,169,000
Total allowance for loan losses	3,050,000	3,050,000
Non-covered loans
Individually evaluated for impairment	11,513,000	19,607,000
Collectively evaluated for impairment	44,735,000	61,791,000
Total non-covered loans	56,248,000	81,398,000
Impaired Loans Receivable, Recorded Investment [Abstract]
With no related allowance recorded	6,255,000	15,251,000
With an allowance recorded	5,258,000	4,356,000
Related Allowance	1,226,000	881,000
Impaired Loans, Unpaid Principal Balance [Abstract]
Loans with no related allowance, unpaid principal balance	9,773,000	23,133,000
Loans with related allowance, unpaid principal balance	5,258,000	4,374,000
Troubled Debt Restructurings by Status [Abstract]
Accrual status	6,788,000	10,397,000
Nonaccrual status	4,664,000	9,013,000
Total modifications	11,452,000	19,410,000
Troubled Debt Restructuring (TDR) [Abstract]
Number of loans restructured (in contracts)	2	3
Pre-modification investment	175,000	1,326,000
Post-modification investment	85,000	1,195,000
Non-covered nonaccrual loans and past due loans, by class
30 - 59 Days past due	59,000	0
60 - 89 Days past due	0	63,000
Greater than 90 days and accruing	0	0
Total past due	59,000	63,000
Nonaccrual	4,725,000	9,210,000
Current	51,464,000	72,125,000
Total non-covered loans	56,248,000	81,398,000
Real Estate Construction [Member] | Pass/Watch [Member]
Non-covered loans
Total non-covered loans	41,397,000	57,555,000
Real Estate Construction [Member] | Special Mention [Member]
Non-covered loans
Total non-covered loans	3,162,000	1,248,000
Real Estate Construction [Member] | Substandard [Member]
Non-covered loans
Total non-covered loans	11,689,000	22,595,000
Real Estate Construction [Member] | Doubtful [Member]
Non-covered loans
Total non-covered loans	0	0
Real Estate Construction [Member] | One to Four Family Residential [Member]
Non-covered loans
Total non-covered loans	33,928,000	46,472,000
Impaired Loans Receivable, Recorded Investment [Abstract]
With no related allowance recorded	6,255,000	15,251,000
With an allowance recorded	5,010,000	4,094,000
Related Allowance	1,182,000	855,000
Impaired Loans, Unpaid Principal Balance [Abstract]
Loans with no related allowance, unpaid principal balance	9,773,000	23,133,000
Loans with related allowance, unpaid principal balance	5,010,000	4,112,000
Troubled Debt Restructurings by Status [Abstract]
Accrual status	6,540,000	10,135,000
Nonaccrual status	4,664,000	9,013,000
Total modifications	11,204,000	19,148,000
Troubled Debt Restructuring (TDR) [Abstract]
Number of loans restructured (in contracts)	2	2
Pre-modification investment	175,000	1,064,000
Post-modification investment	85,000	933,000
Non-covered nonaccrual loans and past due loans, by class
30 - 59 Days past due	59,000	0
60 - 89 Days past due	0	63,000
Greater than 90 days and accruing	0	0
Total past due	59,000	63,000
Nonaccrual	4,725,000	9,210,000
Current	29,144,000	37,199,000
Total non-covered loans	33,928,000	46,472,000
Real Estate Construction [Member] | One to Four Family Residential [Member] | Pass/Watch [Member]
Non-covered loans
Total non-covered loans	21,173,000	25,389,000
Real Estate Construction [Member] | One to Four Family Residential [Member] | Special Mention [Member]
Non-covered loans
Total non-covered loans	3,162,000	776,000
Real Estate Construction [Member] | One to Four Family Residential [Member] | Substandard [Member]
Non-covered loans
Total non-covered loans	9,593,000	20,307,000
Real Estate Construction [Member] | One to Four Family Residential [Member] | Doubtful [Member]
Non-covered loans
Total non-covered loans	0	0
Real Estate Construction [Member] | Multi Family and Commercial [Member]
Non-covered loans
Total non-covered loans	22,320,000	34,926,000
Impaired Loans Receivable, Recorded Investment [Abstract]
With an allowance recorded	248,000	262,000
Related Allowance	44,000	26,000
Impaired Loans, Unpaid Principal Balance [Abstract]
Loans with related allowance, unpaid principal balance	248,000	262,000
Troubled Debt Restructurings by Status [Abstract]
Accrual status	248,000	262,000
Nonaccrual status	0	0
Total modifications	248,000	262,000
Troubled Debt Restructuring (TDR) [Abstract]
Number of loans restructured (in contracts)		1
Pre-modification investment		262,000
Post-modification investment		262,000
Non-covered nonaccrual loans and past due loans, by class
30 - 59 Days past due	0	0
60 - 89 Days past due	0	0
Greater than 90 days and accruing	0	0
Total past due	0	0
Nonaccrual	0	0
Current	22,320,000	34,926,000
Total non-covered loans	22,320,000	34,926,000
Real Estate Construction [Member] | Multi Family and Commercial [Member] | Pass/Watch [Member]
Non-covered loans
Total non-covered loans	20,224,000	32,166,000
Real Estate Construction [Member] | Multi Family and Commercial [Member] | Special Mention [Member]
Non-covered loans
Total non-covered loans	0	472,000
Real Estate Construction [Member] | Multi Family and Commercial [Member] | Substandard [Member]
Non-covered loans
Total non-covered loans	2,096,000	2,288,000
Real Estate Construction [Member] | Multi Family and Commercial [Member] | Doubtful [Member]
Non-covered loans
Total non-covered loans	0	0
Consumer [Member]
Loans, Allowance for Credit Losses [Roll Forward]
Balance at beginning of period	3,176,000	3,452,000
Provision	1,649,000	749,000
Charge-offs	(2,439,000)	(1,526,000)
Recoveries	507,000	501,000
Balance at end of period	2,893,000	3,176,000
Loans, Allowance for Credit Loss, Additional Information [Abstract]
Individually evaluated for impairment	44,000	52,000
Collectively evaluated for impairment	2,849,000	3,124,000
Total allowance for loan losses	2,893,000	3,176,000
Non-covered loans
Individually evaluated for impairment	439,000	916,000
Collectively evaluated for impairment	161,234,000	153,974,000
Total non-covered loans	161,673,000	154,890,000
Impaired Loans Receivable, Recorded Investment [Abstract]
With no related allowance recorded	118,000	583,000
With an allowance recorded	321,000	333,000
Related Allowance	44,000	52,000
Impaired Loans, Unpaid Principal Balance [Abstract]
Loans with no related allowance, unpaid principal balance	269,000	1,017,000
Loans with related allowance, unpaid principal balance	322,000	333,000
Troubled Debt Restructurings by Status [Abstract]
Accrual status	322,000	332,000
Nonaccrual status	32,000	20,000
Total modifications	354,000	352,000
Troubled Debt Restructuring (TDR) [Abstract]
Number of loans restructured (in contracts)	1
Pre-modification investment	23,000
Post-modification investment	20,000
Non-covered nonaccrual loans and past due loans, by class
30 - 59 Days past due	1,363,000	1,435,000
60 - 89 Days past due	106,000	527,000
Greater than 90 days and accruing	0	0
Total past due	1,469,000	1,962,000
Nonaccrual	118,000	583,000
Current	160,086,000	152,345,000
Total non-covered loans	161,673,000	154,890,000
Consumer [Member] | Pass/Watch [Member]
Non-covered loans
Total non-covered loans	155,162,000	147,252,000
Consumer [Member] | Special Mention [Member]
Non-covered loans
Total non-covered loans	1,520,000	466,000
Consumer [Member] | Substandard [Member]
Non-covered loans
Total non-covered loans	4,991,000	7,172,000
Consumer [Member] | Doubtful [Member]
Non-covered loans
Total non-covered loans	0	0
Consumer [Member] | Indirect Consumer Loan [Member]
Non-covered loans
Total non-covered loans	79,900,000	77,596,000
Non-covered nonaccrual loans and past due loans, by class
30 - 59 Days past due	798,000	966,000
60 - 89 Days past due	101,000	112,000
Greater than 90 days and accruing		0
Total past due	899,000	1,078,000
Nonaccrual	0	0
Current	79,001,000	76,518,000
Total non-covered loans	79,900,000	77,596,000
Consumer [Member] | Indirect Consumer Loan [Member] | Pass/Watch [Member]
Non-covered loans
Total non-covered loans	78,661,000	76,076,000
Consumer [Member] | Indirect Consumer Loan [Member] | Special Mention [Member]
Non-covered loans
Total non-covered loans	11,000	16,000
Consumer [Member] | Indirect Consumer Loan [Member] | Substandard [Member]
Non-covered loans
Total non-covered loans	1,228,000	1,504,000
Consumer [Member] | Indirect Consumer Loan [Member] | Doubtful [Member]
Non-covered loans
Total non-covered loans	0	0
Consumer [Member] | Direct Consumer Loan [Member]
Non-covered loans
Total non-covered loans	81,773,000	77,294,000
Impaired Loans Receivable, Recorded Investment [Abstract]
With no related allowance recorded	118,000	583,000
With an allowance recorded	321,000	333,000
Related Allowance	44,000	52,000
Impaired Loans, Unpaid Principal Balance [Abstract]
Loans with no related allowance, unpaid principal balance	269,000	1,017,000
Loans with related allowance, unpaid principal balance	322,000	333,000
Troubled Debt Restructurings by Status [Abstract]
Accrual status	322,000	332,000
Nonaccrual status	32,000	20,000
Total modifications	354,000	352,000
Troubled Debt Restructuring (TDR) [Abstract]
Number of loans restructured (in contracts)	1
Pre-modification investment	23,000
Post-modification investment	20,000
Non-covered nonaccrual loans and past due loans, by class
30 - 59 Days past due	565,000	469,000
60 - 89 Days past due	5,000	415,000
Greater than 90 days and accruing	0	0
Total past due	570,000	884,000
Nonaccrual	118,000	583,000
Current	81,085,000	75,827,000
Total non-covered loans	81,773,000	77,294,000
Consumer [Member] | Direct Consumer Loan [Member] | Pass/Watch [Member]
Non-covered loans
Total non-covered loans	76,501,000	71,176,000
Consumer [Member] | Direct Consumer Loan [Member] | Special Mention [Member]
Non-covered loans
Total non-covered loans	1,509,000	450,000
Consumer [Member] | Direct Consumer Loan [Member] | Substandard [Member]
Non-covered loans
Total non-covered loans	3,763,000	5,668,000
Consumer [Member] | Direct Consumer Loan [Member] | Doubtful [Member]
Non-covered loans
Total non-covered loans	0	0
Non-accrual Loans [Member]
Impaired Loans Receivable, Recorded Investment [Abstract]
With no related allowance recorded	8,123,000	15,479,000
With an allowance recorded	286,000	72,000
Related Allowance	10,000	5,000
Restructured Loans Member]
Impaired Loans Receivable, Recorded Investment [Abstract]
With no related allowance recorded	4,118,000	8,635,000
With an allowance recorded	20,291,000	17,668,000
Related Allowance	$ 4,396,000	$ 3,550,000