Segment Reporting (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2017	Sep. 30, 2016	Sep. 30, 2017	Sep. 30, 2016	Dec. 31, 2016	Dec. 31, 2015
Revenues:
Interest income	$ 14,630	$ 13,523	$ 41,512	$ 40,485
Mortgage fee income	10,018	10,668	28,604	26,850
Insurance and investment services income	147	128	395	303
Other income	(7)	1,730	1,550	5,986
Total operating income	24,788	26,049	72,061	73,624
Expenses:
Interest expense	3,216	2,794	8,898	8,319
Salary and employee benefits	11,249	12,383	33,009	34,427
Provision for loan losses	96	1,075	1,137	2,975
Other expense	6,717	6,356	19,777	17,968
Total operating expenses	21,278	22,608	62,821	63,689
Income from continuing operations, before income taxes	3,510	3,441	9,240	9,935
Income tax (benefit) - continuing operations	1,192	1,131	3,088	3,265
Net income from continuing operations	2,318	2,310	6,152	6,670
Income from discontinued operations, before income taxes	0	0	0	6,346
Income tax benefit - discontinued operations	0	0	0	2,411
Net income from discontinued operations	0	0	0	3,935
Net income	2,318	2,310	6,152	10,605
Preferred dividends	123	314	374	814
Net income available to common shareholders	2,195	1,996	5,778	9,791
Capital expenditures	389	161	4,008	1,435
Total assets	1,471,590	1,468,595	1,471,590	1,468,595	$ 1,418,804	$ 1,384,476
Goodwill	18,480	18,480	18,480	18,480	18,480	18,480
Operating Segments | Commercial & Retail Banking
Revenues:
Interest income	13,432	12,528	38,651	37,583
Mortgage fee income	200	(95)	573	(190)
Insurance and investment services income	147	128	395	303
Other income	1,319	1,687	3,679	4,161
Total operating income	15,098	14,248	43,298	41,857
Expenses:
Interest expense	2,347	2,113	6,635	6,312
Salary and employee benefits	3,107	2,924	9,030	8,654
Provision for loan losses	0	1,075	966	2,975
Other expense	4,822	4,782	14,539	13,686
Total operating expenses	10,276	10,894	31,170	31,627
Income from continuing operations, before income taxes	4,822	3,354	12,128	10,230
Income tax (benefit) - continuing operations	1,605	1,027	3,931	3,177
Net income from continuing operations	3,217	2,327	8,197	7,053
Income from discontinued operations, before income taxes		0	0	0
Income tax benefit - discontinued operations		0	0	0
Net income from discontinued operations		0	0	0
Net income	3,217	2,327	8,197	7,053
Preferred dividends	0	0	0
Net income available to common shareholders	3,217	2,327	8,197	7,053
Capital expenditures	109	116	2,709	1,068
Total assets	1,466,845	1,462,496	1,466,845	1,462,496	1,415,735	1,378,988
Goodwill	1,598	1,598	1,598	1,598	1,598	1,598
Operating Segments | Mortgage Banking
Revenues:
Interest income	1,352	1,183	3,206	3,278
Mortgage fee income	10,042	11,003	28,616	27,862
Insurance and investment services income	0	0	0	0
Other income	(1,279)	(31)	(1,973)	1,804
Total operating income	10,115	12,155	29,849	32,944
Expenses:
Interest expense	684	552	1,521	1,543
Salary and employee benefits	6,768	7,724	19,870	20,866
Provision for loan losses	96	0	171	0
Other expense	2,100	2,054	6,244	5,979
Total operating expenses	9,648	10,330	27,806	28,388
Income from continuing operations, before income taxes	467	1,825	2,043	4,556
Income tax (benefit) - continuing operations	191	704	827	1,762
Net income from continuing operations	276	1,121	1,216	2,794
Income from discontinued operations, before income taxes		0	0	0
Income tax benefit - discontinued operations		0	0	0
Net income from discontinued operations		0	0	0
Net income	276	1,121	1,216	2,794
Preferred dividends	0	0	0
Net income available to common shareholders	276	1,121	1,216	2,794
Capital expenditures	129	9	1,102	164
Total assets	140,954	150,091	140,954	150,091	122,242	125,227
Goodwill	16,882	16,882	16,882	16,882	16,882	16,882
Operating Segments | Financial Holding Company
Revenues:
Interest income	1	1	3	2
Mortgage fee income	0	0	0	0
Insurance and investment services income	0	0	0	0
Other income	1,250	1,444	3,768	4,310
Total operating income	1,251	1,445	3,771	4,312
Expenses:
Interest expense	565	560	1,674	1,665
Salary and employee benefits	1,374	1,735	4,109	4,907
Provision for loan losses	0	0	0	0
Other expense	1,091	888	2,919	2,591
Total operating expenses	3,030	3,183	8,702	9,163
Income from continuing operations, before income taxes	(1,779)	(1,738)	(4,931)	(4,851)
Income tax (benefit) - continuing operations	(604)	(600)	(1,670)	(1,674)
Net income from continuing operations	(1,175)	(1,138)	(3,261)	(3,177)
Income from discontinued operations, before income taxes		0	0	6,926
Income tax benefit - discontinued operations		0	0	2,629
Net income from discontinued operations		0	0	4,297
Net income	(1,175)	(1,138)	(3,261)	1,120
Preferred dividends	123	314	374	814
Net income available to common shareholders	(1,298)	(1,452)	(3,635)	306
Capital expenditures	151	36	197	203
Total assets	183,231	159,753	183,231	159,753	180,335	151,441
Goodwill	0	0	0	0	0	0
Operating Segments | Insurance
Revenues:
Interest income		0		0
Mortgage fee income		0		0
Insurance and investment services income		0		0
Other income		0		0
Total operating income		0		0
Expenses:
Interest expense		0		0
Salary and employee benefits		0		0
Provision for loan losses		0		0
Other expense		0		0
Total operating expenses		0		0
Income from continuing operations, before income taxes		0		0
Income tax (benefit) - continuing operations		0		0
Net income from continuing operations		0		0
Income from discontinued operations, before income taxes		0		(580)
Income tax benefit - discontinued operations		0		(218)
Net income from discontinued operations		0		(362)
Net income		0		(362)
Preferred dividends		0
Net income available to common shareholders		0		(362)
Capital expenditures		0		0
Total assets		0		0		5,017
Goodwill		0		0		0
Intercompany Eliminations
Revenues:
Interest income	(155)	(189)	(348)	(378)
Mortgage fee income	(224)	(240)	(585)	(822)
Insurance and investment services income	0	0	0	0
Other income	(1,297)	(1,370)	(3,924)	(4,289)
Total operating income	(1,676)	(1,799)	(4,857)	(5,489)
Expenses:
Interest expense	(380)	(431)	(932)	(1,201)
Salary and employee benefits	0	0	0	0
Provision for loan losses	0	0	0	0
Other expense	(1,296)	(1,368)	(3,925)	(4,288)
Total operating expenses	(1,676)	(1,799)	(4,857)	(5,489)
Income from continuing operations, before income taxes	0	0	0	0
Income tax (benefit) - continuing operations	0	0	0	0
Net income from continuing operations	0	0	0	0
Income from discontinued operations, before income taxes		0	0	0
Income tax benefit - discontinued operations		0	0	0
Net income from discontinued operations		0	0	0
Net income	0	0	0	0
Preferred dividends	0	0	0
Net income available to common shareholders	0	0	0	0
Capital expenditures	0	0	0	0
Total assets	(319,440)	(303,745)	(319,440)	(303,745)	(299,508)	(276,197)
Goodwill	$ 0	$ 0	$ 0	$ 0	$ 0	$ 0