Segment Reporting - Narrative (Details) $ in Thousands		3 Months Ended		9 Months Ended
	Jun. 30, 2016 USD ($)	Sep. 30, 2016 USD ($)	Sep. 30, 2015 USD ($)	Sep. 30, 2016 USD ($) segment	Sep. 30, 2015 USD ($)
Segment reporting information
Number of reportable segments | segment				4
Net income		$ 2,310	$ 1,406	$ 10,605	$ 5,414
Salary and employee benefits		12,383	9,284	34,427	27,424
Net income (loss) from continuing operations		2,310	1,510	6,670	5,349
Commercial & Retail Banking
Segment reporting information
Net income		2,300	1,600	7,100	3,800
Increase (decrease) in net interest income		1,800		7,100
Increase (decrease) in average loan balances		164,900		207,900
Increase (decrease) in average interest bearing liabilities		154,400		200,700
Increase (decrease) in interest expense		335		1,400
Increase (decrease) in noninterest income		804		1,300
Increase (decrease) in gain on sale of securities		475		882
Increase (decrease) in derivative valuation		(528)		644
Increase (decrease) in noninterest expense		980		2,200
Increase (decrease) in other operating expense		852		1,200
Increase (decrease) in occupancy and equipment		81		410
Salary and employee benefits		29
Increase in data processing and communications expense		350
Increase (decrease) in noninterest expense, intercompany services expense		289		681
Increase (decrease) in income tax expense		460		1,900
Mortgage Banking
Segment reporting information
Net income		1,100	$ 719	2,800	$ 3,700
Increase (decrease) in net interest income		(17)		58
Increase (decrease) in noninterest income		3,200		3,900
Increase (decrease) in derivative valuation		1,500		721
Increase (decrease) in noninterest expense		2,600		5,400
Increase (decrease) in earnings		402		(889)
Increase (decrease) in income tax expense		$ 256		$ (525)
Increase (decrease) loan origination volume (as a percent)		10.00%	(9.00%)	71.00%	58.00%
Decrease in loans held for sale		$ 123,100	$ 73,000	$ 123,100	$ 73,000
Increase (decrease) in mortgage fee income		1,700		3,200
Increase (decrease) in personnel expense		2,400		4,900
Financial Holding Company
Segment reporting information
Increase (decrease) in interest expense		5		17
Increase (decrease) in noninterest income		318		991
Increase (decrease) in gain on sale of securities				70
Increase (decrease) in noninterest expense		761		2,500
Increase (decrease) in occupancy and equipment		64		198
Increase (decrease) in income tax expense		154		545
Net income (loss) from continuing operations		(1,100)	(844)	(3,200)	(2,200)
Increase (decrease) in other operating income		318		921
Increase (decrease) in salaries expense		648		1,800
Increase (decrease) in reorganization expense				500
Increase (decrease) in professional fees		39		398
Insurance
Segment reporting information
Net income			$ (104)	(362)	$ 65
Increase (decrease) in income tax expense		$ 63		$ 261
USI Insurance Services
Segment reporting information
Pre-tax gain	$ 6,900