Loans and Allowance for Loan Losses - Allowance for Loan Losses (Details) (USD $)	3 Months Ended
	Mar. 31, 2013	Dec. 31, 2012	Mar. 31, 2012
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	$ 25,149,000		$ 32,848,000
Charge-offs	1,651,000	49,000,000	5,378,000
Recoveries	757,000		242,000
Provision	237,000		2,350,000
Allowance for loan losses, ending balance	24,492,000	25,149,000	30,062,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	2,122,000	1,397,000
Loans individually evaluated for impairment	33,625,000	33,554,000
Reserves for loans collectively evaluated for impairment	22,370,000	23,752,000
Loans collectively evaluated for impairment	1,269,590,000	1,274,818,000
Loans sold during period	79,000,000
Mortgage Loans Sold with Repurchase and Warranties Exposure		9,500,000,000
Reserve for Off-balance Sheet Activities
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves	177,000	182,000
Reserve for Losses on Mortgage Loans Sold
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves	400,000	1,700,000
Charges against reserves	83,198		173,553
Provision expense	(1,300,000)		108,986
Real Estate Loans | Construction Loans
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	4,269,000		8,214,000
Charge-offs	646,000		1,828,000
Recoveries	516,000		107,000
Provision	(105,000)		1,464,000
Allowance for loan losses, ending balance	4,034,000		7,957,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	240,000	191,000
Loans individually evaluated for impairment	8,284,000	8,170,000
Reserves for loans collectively evaluated for impairment	3,794,000	4,078,000
Loans collectively evaluated for impairment	124,916,000	123,811,000
Real Estate Loans | Owner Occupied
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	4,374,000		5,792,000
Charge-offs	10,000		683,000
Recoveries	24,000		7,000
Provision	25,000		218,000
Allowance for loan losses, ending balance	4,413,000		5,334,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	435,000	189,000
Loans individually evaluated for impairment	10,961,000	11,524,000
Reserves for loans collectively evaluated for impairment	3,978,000	4,185,000
Loans collectively evaluated for impairment	345,994,000	331,438,000
Real Estate Loans | Non-owner Occupied
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	3,935,000		4,668,000
Charge-offs	175,000		1,312,000
Recoveries	6,000		1,000
Provision	45,000		369,000
Allowance for loan losses, ending balance	3,811,000		3,726,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	59,000	89,000
Loans individually evaluated for impairment	5,986,000	5,537,000
Reserves for loans collectively evaluated for impairment	3,752,000	3,846,000
Loans collectively evaluated for impairment	199,488,000	205,952,000
Real Estate Loans | One to Four Family
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	3,191,000		3,726,000
Charge-offs	251,000		375,000
Recoveries	21,000		11,000
Provision	(83,000)		(9,000)
Allowance for loan losses, ending balance	2,878,000		3,353,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	19,000	0
Loans individually evaluated for impairment	1,301,000	1,435,000
Reserves for loans collectively evaluated for impairment	2,859,000	3,191,000
Loans collectively evaluated for impairment	165,264,000	167,176,000
Real Estate Loans | Multifamily
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	594,000		805,000
Charge-offs	0		202,000
Recoveries	0		0
Provision	(15,000)		(97,000)
Allowance for loan losses, ending balance	579,000		506,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	0	0
Loans individually evaluated for impairment	259,000	530,000
Reserves for loans collectively evaluated for impairment	579,000	594,000
Loans collectively evaluated for impairment	34,102,000	34,807,000
Real Estate Loans | Home Equity Lines of Credit
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	3,822,000		3,310,000
Charge-offs	81,000		471,000
Recoveries	84,000		10,000
Provision	309,000		250,000
Allowance for loan losses, ending balance	4,134,000		3,099,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	570,000	0
Loans individually evaluated for impairment	1,207,000	705,000
Reserves for loans collectively evaluated for impairment	3,564,000	3,822,000
Loans collectively evaluated for impairment	192,197,000	191,183,000
Commercial
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	4,291,000		5,712,000
Charge-offs	399,000		425,000
Recoveries	86,000		85,000
Provision	27,000		142,000
Allowance for loan losses, ending balance	4,005,000		5,514,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	799,000	928,000
Loans individually evaluated for impairment	5,627,000	5,653,000
Reserves for loans collectively evaluated for impairment	3,206,000	3,363,000
Loans collectively evaluated for impairment	157,698,000	168,787,000
Consumer and other
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	673,000		621,000
Charge-offs	89,000		82,000
Recoveries	20,000		21,000
Provision	34,000		13,000
Allowance for loan losses, ending balance	638,000		573,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	0	0
Loans individually evaluated for impairment	0	0
Reserves for loans collectively evaluated for impairment	638,000	673,000
Loans collectively evaluated for impairment	$ 49,931,000	$ 51,664,000