Loans (Loan Portfolio by Class According to Credit Quality Indicators) (Details) - USD ($) $ in Thousands	Mar. 31, 2016	Dec. 31, 2015
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	$ 873,766	$ 875,844
SBA, SBA 504, and Commercial Loans [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	533,611	534,264
Small Business Administration Portfolio Segment [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	38,863	39,393
Small Business Administration, 504 Portfolio Segment [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	27,482	29,353
Residential and Consumer Loans [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	340,155	341,580
Commercial Other Receivable [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	50,276	49,332
Commercial Real Estate Portfolio Segment [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	390,345	391,071
Commercial Real Estate Construction Financing Receivable [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	26,645	25,115
Commercial Portfolio Segment [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	467,266	465,518
Residential Portfolio Segment [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	260,957	264,523
Consumer Home Equity Line Financing Receivable [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	44,573	45,042
Consumer Other Financing Receivable [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	34,625	32,015
Consumer Portfolio Segment [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	79,198	77,057
Pass [Member] | SBA, SBA 504, and Commercial Loans [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	502,572	498,476
Pass [Member] | Small Business Administration Portfolio Segment [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	35,008	35,032
Pass [Member] | Small Business Administration, 504 Portfolio Segment [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	23,883	24,003
Pass [Member] | Commercial Other Receivable [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	46,719	45,870
Pass [Member] | Commercial Real Estate Portfolio Segment [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	371,371	369,510
Pass [Member] | Commercial Real Estate Construction Financing Receivable [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	25,591	24,061
Pass [Member] | Commercial Portfolio Segment [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	443,681	439,441
Special Mention [Member] | SBA, SBA 504, and Commercial Loans [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	26,388	29,969
Special Mention [Member] | Small Business Administration Portfolio Segment [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	2,199	2,647
Special Mention [Member] | Small Business Administration, 504 Portfolio Segment [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	2,935	4,917
Special Mention [Member] | Commercial Other Receivable [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	2,493	2,373
Special Mention [Member] | Commercial Real Estate Portfolio Segment [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	18,063	18,978
Special Mention [Member] | Commercial Real Estate Construction Financing Receivable [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	698	1,054
Special Mention [Member] | Commercial Portfolio Segment [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	21,254	22,405
Substandard [Member] | SBA, SBA 504, and Commercial Loans [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	4,651	5,819
Substandard [Member] | Small Business Administration Portfolio Segment [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,656	1,714
Substandard [Member] | Small Business Administration, 504 Portfolio Segment [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	664	433
Substandard [Member] | Commercial Other Receivable [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	1,064	1,089
Substandard [Member] | Commercial Real Estate Portfolio Segment [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	911	2,583
Substandard [Member] | Commercial Real Estate Construction Financing Receivable [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	356
Substandard [Member] | Commercial Portfolio Segment [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	2,331	3,672
Performing Financing Receivable [Member] | Residential and Consumer Loans [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	336,578	338,766
Performing Financing Receivable [Member] | Residential Portfolio Segment [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	257,808	262,299
Performing Financing Receivable [Member] | Consumer Home Equity Line Financing Receivable [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	44,145	44,452
Performing Financing Receivable [Member] | Consumer Other Financing Receivable [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	34,625	32,015
Performing Financing Receivable [Member] | Consumer Portfolio Segment [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	78,770	76,467
Nonperforming Financing Receivable [Member] | Residential and Consumer Loans [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	3,577	2,814
Nonperforming Financing Receivable [Member] | Residential Portfolio Segment [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	3,149	2,224
Nonperforming Financing Receivable [Member] | Consumer Home Equity Line Financing Receivable [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	428	590
Nonperforming Financing Receivable [Member] | Consumer Portfolio Segment [Member]
Financing Receivable, Recorded Investment [Line Items]
Total loans held for investment	$ 428	$ 590