Loans and Allowance for Loan Losses - Allowance for Loan Losses (Details) (USD $)	3 Months Ended		6 Months Ended
	Jun. 30, 2014	Jun. 30, 2013	Jun. 30, 2014	Jun. 30, 2013	Dec. 31, 2013
Financing Receivable, Allowance for Credit Losses
Impaired Financing Receivable, Previously Acquired, Carrying Amount, Balance	$ 88,763		$ 88,763		$ 90,741
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	16,522,000	24,492,000	18,063,000	25,149,000
Charge-offs	347,000	2,115,000	2,638,000	3,767,000
Recoveries	1,165,000	492,000	2,375,000	1,250,000
Provision	(891,000)	55,000	(1,351,000)	292,000
Allowance for loan losses, ending balance	16,449,000	22,924,000	16,449,000	22,924,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	1,995,000		1,995,000		557,000
Loans individually evaluated for impairment	22,188,000		22,188,000		24,526,000
Reserves for loans collectively evaluated for impairment	14,454,000		14,454,000		17,506,000
Loans collectively evaluated for impairment	1,396,556,000		1,396,556,000		1,333,241,000
Loans sold during period	36,800,000
Reserve for Off-balance Sheet Activities
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves	109,369		109,369		116,309
Reserve for Losses on Mortgage Loans Sold
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves	202,080		202,080		216,687
Charges against reserves	2,182	0	2,182	83,198
Provision expense	(7,512)	3,898	(12,424)	(1,300,000)
Commercial
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	4,072,000	4,005,000	3,733,000	4,291,000
Charge-offs	81,000	224,000	826,000	622,000
Recoveries	111,000	236,000	430,000	321,000
Provision	606,000	(32,000)	1,371,000	(5,000)
Allowance for loan losses, ending balance	4,708,000	3,985,000	4,708,000	3,985,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	908,000		908,000		105,000
Loans individually evaluated for impairment	2,386,000		2,386,000		4,403,000
Reserves for loans collectively evaluated for impairment	3,801,000		3,801,000		3,628,000
Loans collectively evaluated for impairment	182,052,000		182,052,000		181,040,000
Consumer and other
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	440,000	638,000	531,000	673,000
Charge-offs	90,000	82,000	128,000	172,000
Recoveries	56,000	28,000	76,000	49,000
Provision	(8,000)	71,000	(81,000)	105,000
Allowance for loan losses, ending balance	398,000	655,000	398,000	655,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	0		0		0
Loans individually evaluated for impairment	0		0		0
Reserves for loans collectively evaluated for impairment	398,000		398,000		531,000
Loans collectively evaluated for impairment	45,975,000		45,975,000		51,667,000
Construction Loans
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	2,403,000	4,034,000	3,169,000	4,269,000
Charge-offs	69,000	1,188,000	326,000	1,834,000
Recoveries	71,000	82,000	163,000	598,000
Provision	(100,000)	737,000	(701,000)	632,000
Allowance for loan losses, ending balance	2,305,000	3,665,000	2,305,000	3,665,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	191,000		191,000		40,000
Loans individually evaluated for impairment	1,980,000		1,980,000		3,016,000
Reserves for loans collectively evaluated for impairment	2,115,000		2,115,000		3,129,000
Loans collectively evaluated for impairment	125,573,000		125,573,000		128,019,000
Owner Occupied
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	2,988,000	4,413,000	3,089,000	4,374,000
Charge-offs	0	234,000	0	244,000
Recoveries	763,000	18,000	784,000	42,000
Provision	(903,000)	(37,000)	(1,025,000)	(12,000)
Allowance for loan losses, ending balance	2,848,000	4,160,000	2,848,000	4,160,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	134,000		134,000		201,000
Loans individually evaluated for impairment	13,306,000		13,306,000		10,576,000
Reserves for loans collectively evaluated for impairment	2,714,000		2,714,000		2,888,000
Loans collectively evaluated for impairment	404,034,000		404,034,000		372,190,000
Non-owner Occupied
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	1,971,000	3,811,000	2,081,000	3,935,000
Charge-offs	80,000	0	889,000	175,000
Recoveries	116,000	18,000	697,000	24,000
Provision	(412,000)	(370,000)	(294,000)	(325,000)
Allowance for loan losses, ending balance	1,595,000	3,459,000	1,595,000	3,459,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	42,000		42,000		50,000
Loans individually evaluated for impairment	2,112,000		2,112,000		5,114,000
Reserves for loans collectively evaluated for impairment	1,552,000		1,552,000		2,031,000
Loans collectively evaluated for impairment	201,138,000		201,138,000		191,812,000
One to Four Family
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	2,111,000	2,878,000	2,368,000	3,191,000
Charge-offs	8,000	387,000	240,000	639,000
Recoveries	8,000	79,000	15,000	100,000
Provision	392,000	74,000	360,000	(8,000)
Allowance for loan losses, ending balance	2,503,000	2,644,000	2,503,000	2,644,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	720,000		720,000		161,000
Loans individually evaluated for impairment	2,404,000		2,404,000		992,000
Reserves for loans collectively evaluated for impairment	1,783,000		1,783,000		2,207,000
Loans collectively evaluated for impairment	175,884,000		175,884,000		173,080,000
Multifamily
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	429,000	579,000	436,000	594,000
Charge-offs	0	0	80,000	0
Recoveries	0	0	0	0
Provision	(57,000)	(66,000)	16,000	(81,000)
Allowance for loan losses, ending balance	372,000	513,000	372,000	513,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	0		0		0
Loans individually evaluated for impairment	0		0		0
Reserves for loans collectively evaluated for impairment	372,000		372,000		436,000
Loans collectively evaluated for impairment	64,805,000		64,805,000		41,713,000
Home Equity Lines of Credit
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Allowance for loan losses, beginning balance	2,108,000	4,134,000	2,656,000	3,822,000
Charge-offs	19,000	0	149,000	81,000
Recoveries	40,000	31,000	210,000	116,000
Provision	(409,000)	(322,000)	(997,000)	(14,000)
Allowance for loan losses, ending balance	1,720,000	3,843,000	1,720,000	3,843,000
Financing Receivable, Allowance for Credit Loss, Additional Information
Reserves for loans individually evaluated for impairment	0		0		0
Loans individually evaluated for impairment	0		0		425,000
Reserves for loans collectively evaluated for impairment	1,719,000		1,719,000		2,656,000
Loans collectively evaluated for impairment	$ 197,095,000		$ 197,095,000		$ 193,720,000