Loans and Allowance for Loan Losses - Allowance for Loan Losses (Details) (USD $)	3 Months Ended		9 Months Ended
	Sep. 30, 2013	Sep. 30, 2012	Sep. 30, 2013	Sep. 30, 2012	Dec. 31, 2012
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	$ 22,924,000	$ 28,797,000	$ 25,149,000	$ 32,848,000
Charge-offs	2,815,000	6,237,000	6,582,000	16,524,000
Recoveries	865,000	420,000	2,115,000	2,087,000
Provision	40,000	4,251,000	332,000	8,820,000
Allowance for loan losses, ending balance	21,014,000	27,231,000	21,014,000	27,231,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	409,000		409,000		1,397,000
Loans individually evaluated for impairment	27,540,000		27,540,000		33,554,000
Reserves for loans collectively evaluated for impairment	20,605,000		20,605,000		23,752,000
Loans collectively evaluated for impairment	1,304,956,000		1,304,956,000		1,274,818,000
Loans sold during period	65,100,000		214,700,000
Mortgage Loans Sold with Repurchase and Warranties Exposure					9,500,000,000
Reserve for Off-balance Sheet Activities
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves	193,385		193,385		182,000
Reserve for Losses on Mortgage Loans Sold
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves	365,211		365,211		1,700,000
Charges against reserves	0	115,238	83,198	(157,868)
Provision expense	6,621	204,861	(1,300,000)	234,576
Real Estate Loans | Construction Loans
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	3,665,000	5,903,000	4,269,000	8,214,000
Charge-offs	254,000	3,881,000	2,088,000	7,823,000
Recoveries	128,000	45,000	726,000	323,000
Provision	(37,000)	3,675,000	595,000	5,028,000
Allowance for loan losses, ending balance	3,502,000	5,742,000	3,502,000	5,742,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	0		0		191,000
Loans individually evaluated for impairment	3,990,000		3,990,000		8,170,000
Reserves for loans collectively evaluated for impairment	3,501,000		3,501,000		4,078,000
Loans collectively evaluated for impairment	124,960,000		124,960,000		123,811,000
Real Estate Loans | Owner Occupied
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	4,160,000	5,630,000	4,374,000	5,792,000
Charge-offs	139,000	146,000	383,000	2,019,000
Recoveries	39,000	2,000	80,000	126,000
Provision	(189,000)	(943,000)	(200,000)	644,000
Allowance for loan losses, ending balance	3,871,000	4,543,000	3,871,000	4,543,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	113,000		113,000		189,000
Loans individually evaluated for impairment	11,643,000		11,643,000		11,524,000
Reserves for loans collectively evaluated for impairment	3,758,000		3,758,000		4,185,000
Loans collectively evaluated for impairment	362,734,000		362,734,000		331,438,000
Real Estate Loans | Non-owner Occupied
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	3,459,000	3,582,000	3,935,000	4,668,000
Charge-offs	0	697,000	175,000	2,461,000
Recoveries	60,000	256,000	85,000	289,000
Provision	(648,000)	1,096,000	(974,000)	1,741,000
Allowance for loan losses, ending balance	2,871,000	4,237,000	2,871,000	4,237,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	0		0		89,000
Loans individually evaluated for impairment	5,490,000		5,490,000		5,537,000
Reserves for loans collectively evaluated for impairment	2,871,000		2,871,000		3,846,000
Loans collectively evaluated for impairment	195,587,000		195,587,000		205,952,000
Real Estate Loans | One to Four Family
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	2,644,000	3,063,000	3,191,000	3,726,000
Charge-offs	49,000	57,000	687,000	625,000
Recoveries	126,000	4,000	226,000	153,000
Provision	267,000	(83,000)	258,000	(327,000)
Allowance for loan losses, ending balance	2,988,000	2,927,000	2,988,000	2,927,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	296,000		296,000		0
Loans individually evaluated for impairment	1,320,000		1,320,000		1,435,000
Reserves for loans collectively evaluated for impairment	2,692,000		2,692,000		3,191,000
Loans collectively evaluated for impairment	170,427,000		170,427,000		167,176,000
Real Estate Loans | Multifamily
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	513,000	563,000	594,000	805,000
Charge-offs	0	0	0	213,000
Recoveries	0	0	0	0
Provision	28,000	5,000	(53,000)	(24,000)
Allowance for loan losses, ending balance	541,000	568,000	541,000	568,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	0		0		0
Loans individually evaluated for impairment	249,000		249,000		530,000
Reserves for loans collectively evaluated for impairment	542,000		542,000		594,000
Loans collectively evaluated for impairment	40,413,000		40,413,000		34,807,000
Real Estate Loans | Home Equity Lines of Credit
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	3,843,000	4,399,000	3,822,000	3,310,000
Charge-offs	803,000	372,000	884,000	1,073,000
Recoveries	261,000	29,000	376,000	50,000
Provision	5,000	(162,000)	(8,000)	1,607,000
Allowance for loan losses, ending balance	3,306,000	3,894,000	3,306,000	3,894,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	0		0		0
Loans individually evaluated for impairment	429,000		429,000		705,000
Reserves for loans collectively evaluated for impairment	3,306,000		3,306,000		3,822,000
Loans collectively evaluated for impairment	192,870,000		192,870,000		191,183,000
Commercial
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	3,985,000	4,715,000	4,291,000	5,712,000
Charge-offs	1,456,000	771,000	2,079,000	1,789,000
Recoveries	221,000	63,000	543,000	829,000
Provision	494,000	624,000	489,000	(121,000)
Allowance for loan losses, ending balance	3,244,000	4,631,000	3,244,000	4,631,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	0		0		928,000
Loans individually evaluated for impairment	4,419,000		4,419,000		5,653,000
Reserves for loans collectively evaluated for impairment	3,244,000		3,244,000		3,363,000
Loans collectively evaluated for impairment	166,612,000		166,612,000		168,787,000
Consumer and other
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	655,000	942,000	673,000	621,000
Charge-offs	114,000	313,000	286,000	521,000
Recoveries	30,000	21,000	79,000	317,000
Provision	120,000	39,000	225,000	272,000
Allowance for loan losses, ending balance	691,000	689,000	691,000	689,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	0		0		0
Loans individually evaluated for impairment	0		0		0
Reserves for loans collectively evaluated for impairment	691,000		691,000		673,000
Loans collectively evaluated for impairment	$ 51,353,000		$ 51,353,000		$ 51,664,000