Financial Instruments - Carrying Amount and Fair Value of Assets and Liabilities (Details) (USD $) In Thousands, unless otherwise specified	Dec. 31, 2013	Dec. 31, 2012
Level 1
Financial assets:
Cash and cash equivalents	$ 40,995	$ 138,396
Investment securities	3,151	133
Loans and loans held-for-sale, net	0	0
Accrued interest receivable	0	0
Federal Home Loan Bank stock	3,473	4,154
Financial liabilities:
Demand deposits, NOW, savings and money market accounts	0	0
Time deposits	0	0
Borrowed funds	0	0
Accrued interest payable	0	0
Level 1 | Forward Sales Commitments
Financial assets:
Derivative assets	0
Level 1 | Interest Rate Swap Agreements
Financial assets:
Derivative assets	0	0
Financial liabilities:
Derivative liabilities	0	0
Level 1 | Interest Rate Lock Commitments
Financial liabilities:
Derivative liabilities	0
Level 2
Financial assets:
Cash and cash equivalents	0	0
Investment securities	285,771	342,987
Loans and loans held-for-sale, net	18,913	27,679
Accrued interest receivable	1,705	1,760
Federal Home Loan Bank stock	0	0
Financial liabilities:
Demand deposits, NOW, savings and money market accounts	911,457	898,372
Time deposits	560,008	742,552
Borrowed funds	66,690	105,926
Accrued interest payable	852	1,629
Level 2 | Forward Sales Commitments
Financial assets:
Derivative assets	147
Level 2 | Interest Rate Swap Agreements
Financial assets:
Derivative assets	0	0
Financial liabilities:
Derivative liabilities	0	0
Level 2 | Interest Rate Lock Commitments
Financial liabilities:
Derivative liabilities	0	481
Level 3
Financial assets:
Cash and cash equivalents	0	0
Investment securities	0	0
Loans and loans held-for-sale, net	1,275,741	1,159,984
Accrued interest receivable	4,514	4,616
Federal Home Loan Bank stock	0	0
Financial liabilities:
Demand deposits, NOW, savings and money market accounts	0	0
Time deposits	0	0
Borrowed funds	0	0
Accrued interest payable	0	0
Level 3 | Forward Sales Commitments
Financial assets:
Derivative assets	0
Level 3 | Interest Rate Swap Agreements
Financial assets:
Derivative assets	132	196
Financial liabilities:
Derivative liabilities	132	196
Level 3 | Interest Rate Lock Commitments
Financial liabilities:
Derivative liabilities	23
Carrying Amount
Financial assets:
Cash and cash equivalents	40,995	138,396
Investment securities	288,922	343,120
Loans and loans held-for-sale, net	1,359,596	1,312,034
Accrued interest receivable	6,219	6,376
Federal Home Loan Bank stock	3,473	4,154
Financial liabilities:
Demand deposits, NOW, savings and money market accounts	961,154	898,372
Time deposits	557,269	733,290
Borrowed funds	89,214	105,136
Accrued interest payable	852	1,629
Carrying Amount | Forward Sales Commitments
Financial assets:
Derivative assets	147
Carrying Amount | Interest Rate Swap Agreements
Financial assets:
Derivative assets	132	196
Financial liabilities:
Derivative liabilities	132	196
Carrying Amount | Interest Rate Lock Commitments
Financial liabilities:
Derivative liabilities	23
Estimated Fair Value
Financial assets:
Cash and cash equivalents	40,995	138,396
Investment securities	288,922	343,120
Loans and loans held-for-sale, net	1,294,654	1,187,663
Accrued interest receivable	6,219	6,376
Federal Home Loan Bank stock	3,473	4,154
Financial liabilities:
Demand deposits, NOW, savings and money market accounts	911,457	898,372
Time deposits	560,008	742,552
Borrowed funds	66,690	105,926
Accrued interest payable	852	1,629
Estimated Fair Value | Forward Sales Commitments
Financial assets:
Derivative assets	147
Estimated Fair Value | Interest Rate Swap Agreements
Financial assets:
Derivative assets	132	196
Financial liabilities:
Derivative liabilities	132	196
Estimated Fair Value | Interest Rate Lock Commitments
Financial liabilities:
Derivative liabilities	$ 23