Segment Reporting (Details) $ in Thousands	3 Months Ended		6 Months Ended		12 Months Ended
	Jun. 30, 2016 USD ($)	Jun. 30, 2015 USD ($)	Jun. 30, 2016 USD ($) item	Jun. 30, 2015 USD ($)	Dec. 31, 2015 USD ($)	Dec. 31, 2014 USD ($)
Information about the reportable segments and reconciliation to the consolidated financial statements
Number of Reportable Segments | item			4
Revenues:
Interest income	$ 13,580	$ 10,694	$ 26,962	$ 20,332
Mortgage fee income	9,397	8,617	16,182	14,926
Insurance and investment services income	122	107	175	178
Other income	2,481	923	4,254	4,316
Total operating income	25,580	20,341	47,573	39,752
Expenses:
Interest expense	2,838	2,268	5,525	4,359
Provision for loan losses	1,275	561	1,900	1,220
Salary and employee benefits	11,735	9,292	22,044	18,140
Other expense	6,020	5,253	11,610	10,288
Total operating expenses	21,868	17,374	41,079	34,007
Income from continuing operations, before income taxes	3,712	2,967	6,494	5,745
Income tax expense (benefit)	1,254	888	2,134	1,906
Net Income from continuing operations	2,458	2,079	4,360	3,839
Income from discontinued operations, before income taxes	6,516	(279)	6,346	275
Income tax expense - discontinued operations	2,475	(105)	2,411	106
Net Income from discontinued operations	4,041	(174)	3,935	169
Net Income (loss)	6,499	1,905	8,295	4,008
Preferred dividends	314	143	500	285
Net Income available to common shareholders	6,185	1,762	7,795	3,723		$ 1,762
Capital expenditures	624	711	1,274	1,105
Total assets	1,484,962	1,253,066	1,484,962	1,253,066	$ 1,384,476	1,110,459
Goodwill	18,480	17,779	18,480	17,779	18,480	17,779
Gain on sale of subsidiary			6,926
USI Insurance Services [Member]
Revenues:
Pre-tax gain			6,900
Commercial & Retail Banking
Expenses:
Salary and employee benefits	236
Net Income (loss)	2,300	1,700	4,700	2,200
Increase (decrease) in net interest income	2,400		5,400
Increase (decrease) in average loan balances	233,200		229,900
Increase (decrease) in noninterest income	198		474
Increase (decrease) in gain on sale of securities	218		407
Increase (decrease) in noninterest expense	855		1,300
Increase (decrease) in other operating expense	176		367
Increase (decrease) in occupancy and equipment expense	166		329
Increase (decrease) in income tax expense	428		1,400
Increase (decrease) in average subordinated debt	228,400		224,200
Increase (decrease) in derivative valuation	47		115
Increase in data processing and communications expense	284
Increase (decrease) in interest expense	515		1,100
Mortgage Banking
Expenses:
Net Income (loss)	1,300	$ 1,200	1,700		3,000
Increase (decrease) in net interest income	26		41
Increase (decrease) in noninterest income	2,200		709
Increase (decrease) in noninterest expense	1,900		2,800
Increase (decrease) in income tax expense	77			$ (780)
Increase (decrease) in earnings	109		1,300
Increase (decrease) in mortgage fee income	917		1,500
Increase (decrease) in gain on derivative	$ 1,200		$ (789)
Increase (decrease) loan origination volume (as a percent)	3.00%	10.00%	55.00%	73.00%
Increase (decrease) in other operating income			$ 602
Increase (decrease) in mortgage processing expense			118
Increase (decrease) in personnel expense	$ 1,900		2,500
Decrease in loans held for sale	131,700	$ 116,300	131,700	$ 116,300
Financial Holding Company
Revenues:
Interest income	1		1	1
Other income	1,252	1,035	2,866	2,193
Total operating income	1,253	1,035	2,867	2,194
Expenses:
Interest expense	553	549	1,105	1,092
Salary and employee benefits	1,406	1,055	3,172	2,035
Other expense	939	648	1,703	1,083
Total operating expenses	2,898	2,252	5,980	4,210
Income from continuing operations, before income taxes	(1,645)	(1,217)	(3,113)	(2,016)
Income tax expense (benefit)	(558)	(419)	(1,074)	(683)
Net Income from continuing operations	(1,087)	(798)	(2,039)	(1,333)
Income from discontinued operations, before income taxes	6,926		6,926
Income tax expense - discontinued operations	2,629		2,629
Net Income from discontinued operations	4,297		4,297
Net Income (loss)	3,210	(798)	2,258	(1,333)
Preferred dividends	314	143	500	285
Net Income available to common shareholders	2,896		1,758	(1,618)		(941)
Capital expenditures	106	214	167	224
Total assets	160,168	142,260	160,168	142,260	151,441	141,645
Increase (decrease) in noninterest income	217		673
Increase (decrease) in gain on sale of securities			70
Increase (decrease) in noninterest expense	642		1,800
Increase (decrease) in income tax expense	139		391
Increase (decrease) in other operating income	217
Increase (decrease) in reorganization expense			500
Increase (decrease) in professional fees	189		359
Increase (decrease) in salaries expense	351		1,100
Increase (decrease) in interest expense	4		13
Insurance
Expenses:
Net Income (loss)	256	(174)	362	169
Increase (decrease) in income tax expense	49		324
Operating segments | Commercial & Retail Banking
Revenues:
Interest income	12,591	9,718	25,055	18,621
Mortgage fee income	(73)	76	(95)	63
Insurance and investment services income	122	107	175	178
Other income	1,404	1,072	2,474	1,839
Total operating income	14,044	10,973	27,609	20,701
Expenses:
Interest expense	2,159	1,644	4,199	3,142
Provision for loan losses	1,275	561	1,900	1,220
Salary and employee benefits	2,899	2,663	5,730	5,441
Other expense	4,431	3,812	8,904	7,940
Total operating expenses	10,764	8,680	20,733	17,743
Income from continuing operations, before income taxes	3,280	2,293	6,876	2,958
Income tax expense (benefit)	1,012	584	2,150	751
Net Income from continuing operations	2,268	1,709	4,726	2,207
Net Income (loss)	2,268	1,709	4,726	2,207
Net Income available to common shareholders	2,268		4,726	2,207		1,709
Capital expenditures	404	424	952	773
Total assets	1,477,737	1,248,697	1,477,737	1,248,697	1,378,988	1,048,101
Goodwill	1,598	897	1,598	897	1,598	897
Operating segments | Mortgage Banking
Revenues:
Interest income	1,154	1,137	2,095	1,966
Mortgage fee income	9,750	8,833	16,859	15,385
Other income	1,157	(88)	1,835	2,600
Total operating income	12,061	9,882	20,789	19,951
Expenses:
Interest expense	572	529	991	903
Salary and employee benefits	7,430	5,574	13,142	10,664
Other expense	1,982	1,888	3,925	3,581
Total operating expenses	9,984	7,991	18,058	15,148
Income from continuing operations, before income taxes	2,077	1,891	2,731	4,803
Income tax expense (benefit)	800	723	1,058	1,838
Net Income from continuing operations	1,277	1,168	1,673	2,965
Net Income (loss)	1,277	1,168	1,673	2,965
Net Income available to common shareholders	1,277		1,673	2,965		1,168
Capital expenditures	114	71	155	99
Total assets	156,334	139,767	156,334	139,767	125,227	101,791
Goodwill	16,882	16,882	16,882	16,882	16,882	16,882
Operating segments | Insurance
Expenses:
Income from discontinued operations, before income taxes	(410)	(279)	(580)	275
Income tax expense - discontinued operations	(154)	(105)	(218)	106
Net Income from discontinued operations	(256)	(174)	(362)	169
Net Income (loss)	(256)	(174)	(362)	169
Net Income available to common shareholders	(256)		(362)	169		(174)
Capital expenditures		2		9
Total assets		4,321		4,321	5,017	4,031
Intercompany Eliminations
Revenues:
Interest income	(166)	(161)	(189)	(256)
Mortgage fee income	(280)	(292)	(582)	(522)
Other income	(1,332)	(1,096)	(2,921)	(2,316)
Total operating income	(1,778)	(1,549)	(3,692)	(3,094)
Expenses:
Interest expense	(446)	(454)	(770)	(778)
Other expense	(1,332)	(1,095)	(2,922)	(2,316)
Total operating expenses	(1,778)	(1,549)	(3,692)	(3,094)
Total assets	$ (309,277)	$ (281,979)	$ (309,277)	$ (281,979)	$ (276,197)	$ (185,109)