Loans and Allowance for Loan Losses - Nonaccrual and Past Due Loans (Details) (USD $) In Thousands, unless otherwise specified	Dec. 31, 2013 Loans	Dec. 31, 2012 Loans
Financing Receivable, Recorded Investment, Past Due
Financing receivables on nonaccrual status	$ 15,393	$ 22,817
Ratio of nonperforming loans to all loans	1.12%	1.71%
Total number of loans on nonaccrual status	195	315
Past due Loans, Aging Analysis
30 to 59 Days Past Due	9,998	13,943
60 to 89 Days Past Due	2,920	2,000
Greater Than 90 Days	2,486	13,362
Total Past Due	15,404	29,305
Current	1,342,363	1,279,067
Loans	1,357,767	1,308,372
Real Estate Loans | Construction Loans
Financing Receivable, Recorded Investment, Past Due
Financing receivables on nonaccrual status	2,331	7,385
Past due Loans, Aging Analysis
30 to 59 Days Past Due	855	4,395
60 to 89 Days Past Due	207	345
Greater Than 90 Days	541	1,865
Total Past Due	1,603	6,605
Current	129,432	125,376
Loans	131,035	131,981
Real Estate Loans | Owner Occupied Commercial
Financing Receivable, Recorded Investment, Past Due
Financing receivables on nonaccrual status	4,417	5,787
Past due Loans, Aging Analysis
30 to 59 Days Past Due	1,973	838
60 to 89 Days Past Due	0	114
Greater Than 90 Days	58	4,237
Total Past Due	2,031	5,189
Current	380,735	337,773
Loans	382,766	342,962
Real Estate Loans | Non-owner Occupied
Financing Receivable, Recorded Investment, Past Due
Financing receivables on nonaccrual status	1,806	1,697
Past due Loans, Aging Analysis
30 to 59 Days Past Due	1,172	1,688
60 to 89 Days Past Due	129	500
Greater Than 90 Days	432	1,098
Total Past Due	1,733	3,286
Current	195,193	208,203
Loans	196,926	211,489
Real Estate Loans | One to Four Family
Financing Receivable, Recorded Investment, Past Due
Financing receivables on nonaccrual status	2,734	3,123
Past due Loans, Aging Analysis
30 to 59 Days Past Due	2,738	2,767
60 to 89 Days Past Due	1,190	910
Greater Than 90 Days	1,186	2,226
Total Past Due	5,114	5,903
Current	168,958	162,708
Loans	174,072	168,611
Real Estate Loans | Multifamily
Financing Receivable, Recorded Investment, Past Due
Financing receivables on nonaccrual status	256	539
Past due Loans, Aging Analysis
30 to 59 Days Past Due	0	12
60 to 89 Days Past Due	249	0
Greater Than 90 Days	0	10
Total Past Due	249	22
Current	41,464	35,315
Loans	41,713	35,337
Real Estate Loans | Home Equity Lines of Credit
Financing Receivable, Recorded Investment, Past Due
Financing receivables on nonaccrual status	1,285	1,041
Past due Loans, Aging Analysis
30 to 59 Days Past Due	909	1,980
60 to 89 Days Past Due	341	52
Greater Than 90 Days	75	924
Total Past Due	1,325	2,956
Current	192,820	188,932
Loans	194,145	191,888
Commercial
Financing Receivable, Recorded Investment, Past Due
Financing receivables on nonaccrual status	2,306	2,790
Past due Loans, Aging Analysis
30 to 59 Days Past Due	2,029	2,027
60 to 89 Days Past Due	742	34
Greater Than 90 Days	142	2,660
Total Past Due	2,913	4,721
Current	182,530	169,719
Loans	185,443	174,440
Consumer and other
Financing Receivable, Recorded Investment, Past Due
Financing receivables on nonaccrual status	258	455
Past due Loans, Aging Analysis
30 to 59 Days Past Due	322	236
60 to 89 Days Past Due	62	45
Greater Than 90 Days	52	342
Total Past Due	436	623
Current	51,231	51,041
Loans	$ 51,667	$ 51,664