Loans and Allowance for Loan Losses - Allowance for Loan Losses (Details) (USD $)	3 Months Ended
	Mar. 31, 2014	Mar. 31, 2013	Dec. 31, 2013
Financing Receivable, Allowance for Credit Losses
Impaired Financing Receivable, Previously Acquired, Carrying Amount, Balance	$ 89,856		$ 90,741
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	18,063,000	25,149,000
Charge-offs	2,291,000	1,651,000
Recoveries	1,210,000	757,000
Provision	(460,000)	237,000
Allowance for loan losses, ending balance	16,522,000	24,492,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	690,000		557,000
Loans individually evaluated for impairment	20,388,000		24,526,000
Reserves for loans collectively evaluated for impairment	15,831,000		17,506,000
Loans collectively evaluated for impairment	1,361,168,000		1,333,241,000
Loans sold during period	40,000,000	78,700,000
Reserve for Off-balance Sheet Activities
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves	110,339		116,309
Reserve for Losses on Mortgage Loans Sold
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves	211,774		216,687
Charges against reserves	7,030	83,198
Provision expense	(4,914)	(1,300,000)
Commercial
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	3,733,000	4,291,000
Charge-offs	745,000	399,000
Recoveries	319,000	86,000
Provision	765,000	27,000
Allowance for loan losses, ending balance	4,072,000	4,005,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	366,000		105,000
Loans individually evaluated for impairment	1,142,000		4,403,000
Reserves for loans collectively evaluated for impairment	3,706,000		3,628,000
Loans collectively evaluated for impairment	182,683,000		181,040,000
Consumer and other
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	531,000	673,000
Charge-offs	38,000	89,000
Recoveries	20,000	20,000
Provision	(73,000)	34,000
Allowance for loan losses, ending balance	440,000	638,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	0		0
Loans individually evaluated for impairment	0		0
Reserves for loans collectively evaluated for impairment	440,000		531,000
Loans collectively evaluated for impairment	44,687,000		51,667,000
Construction Loans
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	3,169,000	4,269,000
Charge-offs	258,000	646,000
Recoveries	92,000	516,000
Provision	(600,000)	(105,000)
Allowance for loan losses, ending balance	2,403,000	4,034,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	0		40,000
Loans individually evaluated for impairment	3,041,000		3,016,000
Reserves for loans collectively evaluated for impairment	2,403,000		3,129,000
Loans collectively evaluated for impairment	121,905,000		128,019,000
Owner Occupied
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	3,089,000	4,374,000
Charge-offs	0	10,000
Recoveries	22,000	24,000
Provision	(123,000)	25,000
Allowance for loan losses, ending balance	2,988,000	4,413,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	127,000		201,000
Loans individually evaluated for impairment	13,433,000		10,576,000
Reserves for loans collectively evaluated for impairment	2,862,000		2,888,000
Loans collectively evaluated for impairment	376,454,000		372,190,000
Non-owner Occupied
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	2,081,000	3,935,000
Charge-offs	809,000	175,000
Recoveries	581,000	6,000
Provision	118,000	45,000
Allowance for loan losses, ending balance	1,971,000	3,811,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	41,000		50,000
Loans individually evaluated for impairment	1,829,000		5,114,000
Reserves for loans collectively evaluated for impairment	1,929,000		2,031,000
Loans collectively evaluated for impairment	201,143,000		191,812,000
One to Four Family
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	2,368,000	3,191,000
Charge-offs	232,000	251,000
Recoveries	7,000	21,000
Provision	(32,000)	(83,000)
Allowance for loan losses, ending balance	2,111,000	2,878,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	156,000		161,000
Loans individually evaluated for impairment	943,000		992,000
Reserves for loans collectively evaluated for impairment	1,955,000		2,207,000
Loans collectively evaluated for impairment	179,163,000		173,080,000
Multifamily
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	436,000	594,000
Charge-offs	79,000	0
Recoveries	0	0
Provision	72,000	(15,000)
Allowance for loan losses, ending balance	429,000	579,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	0		0
Loans individually evaluated for impairment	0		0
Reserves for loans collectively evaluated for impairment	429,000		436,000
Loans collectively evaluated for impairment	62,332,000		41,713,000
Home Equity Lines of Credit
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	2,656,000	3,822,000
Charge-offs	130,000	81,000
Recoveries	169,000	84,000
Provision	(587,000)	309,000
Allowance for loan losses, ending balance	2,108,000	4,134,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	0		0
Loans individually evaluated for impairment	0		425,000
Reserves for loans collectively evaluated for impairment	2,107,000		2,656,000
Loans collectively evaluated for impairment	$ 192,801,000		$ 193,720,000