Loans and Allowance for Loan Losses - Loans Receivable (Details) (USD $)	3 Months Ended									12 Months Ended
	Dec. 31, 2013	Dec. 31, 2012	Sep. 30, 2012	Jun. 30, 2012	Mar. 31, 2012	Dec. 31, 2011	Sep. 30, 2011	Jun. 30, 2011	Mar. 31, 2011	Dec. 31, 2013	Dec. 31, 2012	Dec. 31, 2011	Dec. 31, 2010
Loans Receivable
Provision for loan losses	$ (3,017,000)		$ 40,000	$ 55,000	$ 237,000	$ 31,554,000	$ 4,251,000	$ 2,218,000	$ 2,351,000	$ (2,685,000)	$ 40,374,000	$ 20,843,000
Loans	1,357,767,000	1,308,372,000								1,357,767,000	1,308,372,000
Less: Net deferred loan origination fees	979,000	1,132,000								979,000	1,132,000
Allowance for loan losses	(18,063,000)	(25,149,000)								(18,063,000)	(25,149,000)
Net loans	1,340,683,000	1,284,355,000								1,340,683,000	1,284,355,000		1,418,076,000
Loans sold during period		43,000,000								254,500,000
Provision		11,000,000								(2,685,000)	40,374,000
Gain (Loss) on Disposition of Assets		35,000,000
Nonperforming Loans
Loans Receivable
Loans sold during period		36,000,000
Real Estate Loans | Construction Loans
Loans Receivable
Loans	131,035,000	131,981,000								131,035,000	131,981,000
Provision										(3,000)	12,742,000
Real Estate Loans | Commercial Real Estate
Loans Receivable
Loans	579,700,000	554,500,000								579,700,000	554,500,000
Real Estate Loans | Owner Occupied Commercial
Loans Receivable
Loans	382,766,000	342,962,000								382,766,000	342,962,000
Provision										(1,022,000)	5,751,000
Real Estate Loans | Non-owner Occupied
Loans Receivable
Loans	196,926,000	211,489,000								196,926,000	211,489,000
Provision										(1,764,000)	8,433,000
Real Estate Loans | One to Four Family
Loans Receivable
Loans	174,072,000	168,611,000								174,072,000	168,611,000
Provision										(205,000)	2,762,000
Real Estate Loans | Multifamily
Loans Receivable
Loans	41,713,000	35,337,000								41,713,000	35,337,000
Provision										(158,000)	136,000
Real Estate Loans | Home Equity Lines of Credit
Loans Receivable
Loans	194,145,000	191,888,000								194,145,000	191,888,000
Provision										(697,000)	4,686,000
Commercial
Loans Receivable
Loans	185,443,000	174,440,000								185,443,000	174,440,000
Provision										1,078,000	5,419,000
Consumer and other
Loans Receivable
Loans	51,667,000	51,664,000								51,667,000	51,664,000
Provision										$ 86,000	$ 445,000