Loans and Allowance for Loan Losses - Allowance for Loan Losses (Details) (USD $)	3 Months Ended			6 Months Ended
	Jun. 30, 2013	Mar. 31, 2013	Jun. 30, 2012	Jun. 30, 2013	Jun. 30, 2012	Dec. 31, 2012
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	$ 24,492,000	$ 25,149,000	$ 30,062,000	$ 25,149,000	$ 32,848,000
Charge-offs	2,115,000		4,907,000	3,767,000	10,287,000
Recoveries	492,000		1,424,000	1,250,000	1,667,000
Provision	54,696		2,218,000	292,000	4,569,000
Allowance for loan losses, ending balance	22,924,000		28,797,000	22,924,000	28,797,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	1,988,000			1,988,000		1,397,000
Loans individually evaluated for impairment	29,441,000			29,441,000		33,554,000
Reserves for loans collectively evaluated for impairment	20,936,000			20,936,000		23,752,000
Loans collectively evaluated for impairment	1,284,306,000			1,284,306,000		1,274,818,000
Loans sold during period	72,000,000			149,600,000
Mortgage Loans Sold with Repurchase and Warranties Exposure						9,500,000,000
Reserve for Off-balance Sheet Activities
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves	185,000			185,000		182,000
Reserve for Losses on Mortgage Loans Sold
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves	358,590			358,590		1,700,000
Charges against reserves	0		208,539	83,198	382,092
Provision expense	(4,990)	0	29,715	0	138,701
Real Estate Loans | Construction Loans
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	4,034,000	4,269,000	7,957,000	4,269,000	8,214,000
Charge-offs	1,188,000		2,114,000	1,834,000	3,942,000
Recoveries	82,000		170,000	598,000	278,000
Provision	737,000		(110,000)	632,000	1,353,000
Allowance for loan losses, ending balance	3,665,000		5,903,000	3,665,000	5,903,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	11,000			11,000		191,000
Loans individually evaluated for impairment	4,901,000			4,901,000		8,170,000
Reserves for loans collectively evaluated for impairment	3,654,000			3,654,000		4,078,000
Loans collectively evaluated for impairment	122,663,000			122,663,000		123,811,000
Real Estate Loans | Owner Occupied
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	4,413,000	4,374,000	5,334,000	4,374,000	5,792,000
Charge-offs	234,000		1,190,000	244,000	1,873,000
Recoveries	18,000		116,000	42,000	124,000
Provision	(37,000)		1,370,000	(12,000)	1,587,000
Allowance for loan losses, ending balance	4,160,000		5,630,000	4,160,000	5,630,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	386,000			386,000		189,000
Loans individually evaluated for impairment	10,869,000			10,869,000		11,524,000
Reserves for loans collectively evaluated for impairment	3,774,000			3,774,000		4,185,000
Loans collectively evaluated for impairment	351,938,000			351,938,000		331,438,000
Real Estate Loans | Non-owner Occupied
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	3,811,000	3,935,000	3,726,000	3,935,000	4,668,000
Charge-offs	0		452,000	175,000	1,764,000
Recoveries	18,000		33,000	24,000	33,000
Provision	(370,000)		275,000	(325,000)	645,000
Allowance for loan losses, ending balance	3,459,000		3,582,000	3,459,000	3,582,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	55,000			55,000		89,000
Loans individually evaluated for impairment	5,588,000			5,588,000		5,537,000
Reserves for loans collectively evaluated for impairment	3,404,000			3,404,000		3,846,000
Loans collectively evaluated for impairment	202,548,000			202,548,000		205,952,000
Real Estate Loans | One to Four Family
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	2,878,000	3,191,000	3,353,000	3,191,000	3,726,000
Charge-offs	387,000		193,000	639,000	568,000
Recoveries	79,000		137,000	100,000	148,000
Provision	74,000		(234,000)	(8,000)	(243,000)
Allowance for loan losses, ending balance	2,644,000		3,063,000	2,644,000	3,063,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	171,000			171,000		0
Loans individually evaluated for impairment	1,023,000			1,023,000		1,435,000
Reserves for loans collectively evaluated for impairment	2,473,000			2,473,000		3,191,000
Loans collectively evaluated for impairment	166,760,000			166,760,000		167,176,000
Real Estate Loans | Multifamily
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	579,000	594,000	506,000	594,000	805,000
Charge-offs	0		11,000	0	213,000
Recoveries	0		0	0	0
Provision	(66,000)		68,000	(81,000)	(29,000)
Allowance for loan losses, ending balance	513,000		563,000	513,000	563,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	0			0		0
Loans individually evaluated for impairment	253,000			253,000		530,000
Reserves for loans collectively evaluated for impairment	513,000			513,000		594,000
Loans collectively evaluated for impairment	33,471,000			33,471,000		34,807,000
Real Estate Loans | Home Equity Lines of Credit
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	4,134,000	3,822,000	3,099,000	3,822,000	3,310,000
Charge-offs	0		229,000	81,000	701,000
Recoveries	31,000		11,000	116,000	21,000
Provision	(322,000)		1,518,000	(14,000)	1,769,000
Allowance for loan losses, ending balance	3,843,000		4,399,000	3,843,000	4,399,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	566,000			566,000		0
Loans individually evaluated for impairment	1,198,000			1,198,000		705,000
Reserves for loans collectively evaluated for impairment	3,277,000			3,277,000		3,822,000
Loans collectively evaluated for impairment	194,450,000			194,450,000		191,183,000
Commercial
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	4,005,000	4,291,000	5,514,000	4,291,000	5,712,000
Charge-offs	224,000		593,000	622,000	1,018,000
Recoveries	236,000		681,000	321,000	766,000
Provision	(32,000)		(887,000)	(5,000)	(745,000)
Allowance for loan losses, ending balance	3,985,000		4,715,000	3,985,000	4,715,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	799,000			799,000		928,000
Loans individually evaluated for impairment	5,609,000			5,609,000		5,653,000
Reserves for loans collectively evaluated for impairment	3,186,000			3,186,000		3,363,000
Loans collectively evaluated for impairment	162,747,000			162,747,000		168,787,000
Consumer and other
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	638,000	673,000	573,000	673,000	621,000
Charge-offs	82,000		125,000	172,000	208,000
Recoveries	28,000		276,000	49,000	297,000
Provision	71,000		218,000	105,000	232,000
Allowance for loan losses, ending balance	655,000		942,000	655,000	942,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	0			0		0
Loans individually evaluated for impairment	0			0		0
Reserves for loans collectively evaluated for impairment	655,000			655,000		673,000
Loans collectively evaluated for impairment	$ 49,729,000			$ 49,729,000		$ 51,664,000