SEGMENT REPORTING - Narrative (Details) $ in Thousands		3 Months Ended								12 Months Ended
	Jun. 30, 2016 USD ($)	Dec. 31, 2016 USD ($)	Sep. 30, 2016 USD ($)	Jun. 30, 2016 USD ($)	Mar. 31, 2016 USD ($)	Dec. 31, 2015 USD ($)	Sep. 30, 2015 USD ($)	Jun. 30, 2015 USD ($)	Mar. 31, 2015 USD ($)	Dec. 31, 2016 USD ($) segment	Dec. 31, 2015 USD ($)	Dec. 31, 2014 USD ($)
Segment Reporting Information [Line Items]
Number of reportable segments | segment										3
Net income (loss)		$ 2,307	$ 2,310	$ 6,499	$ 1,796	$ 1,402	$ 1,406	$ 1,905	$ 2,103	$ 12,912	$ 6,816	$ 2,079
Increase (decrease) in provision expense										3,632	2,493	2,582
Net Income (loss) from continuing operations										8,977	6,603	2,655
Commercial & Retail Banking
Segment Reporting Information [Line Items]
Increase (decrease) in net interest income										8,200
Increase (decrease) in average loan balances										179,000
Increase (decrease) in noninterest income										1,600
Increase (decrease) in performance of interest rate cap										818
Increase (decrease) in gain on sale of securities										882
Increase (decrease) in other operating income										133
Increase (decrease) in debit card income and interchange income										199
Increase (decrease) in volume of portfolio loans sold										(371)
Increase (decrease) in mortgage fee income										(259)
Increase (decrease) in noninterest expense										2,400
Increase (decrease) in salaries expense										543
Increase (decrease) in occupancy and equipment expense										494
Increase (decrease) in data processing expense										832
Increase (decrease) in professional fees										(776)
Increase (decrease) in provision expense										1,100
Mortgage Banking
Segment Reporting Information [Line Items]
Increase (decrease) in net interest income										(32)
Increase (decrease) in noninterest income										6,400
Increase (decrease) in mortgage fee income										6,400
Increase (decrease) in noninterest expense										7,600
Increase (decrease) in salaries expense										6,900
Increase (decrease) in occupancy and equipment expense										117
Increase (decrease) in data processing expense										$ 98
Increase (decrease) in origination volume										26.40%
Increase (decrease) in earn out payments										$ 1,800
Increase (decrease) in mortgage processing expense										197
Increase (decrease) in travel, entertainment, dues, and subscriptions expense										133
Increase (decrease) in other operating expense										134
Increase (decrease) in loan expenses										55
Increase (decrease) in marketing expense										(115)
Financial Holding Company
Segment Reporting Information [Line Items]
Increase (decrease) in net interest income										22
Increase (decrease) in noninterest income										916
Increase (decrease) in noninterest expense										2,300
Increase (decrease) in salaries expense										1,700
Increase (decrease) in occupancy and equipment expense										259
Increase (decrease) in other operating expense										66
Increase (decrease) in income tax expense										475
Increase (decrease) in professional fees										220
Operating Segments | Commercial & Retail Banking
Segment Reporting Information [Line Items]
Net income (loss)										9,900	5,964	4,831
Increase (decrease) in provision expense										3,632	2,493	2,582
Net Income (loss) from continuing operations										9,900	5,964	4,831
Operating Segments | Mortgage Banking
Segment Reporting Information [Line Items]
Net income (loss)										3,062	3,829	(149)
Increase (decrease) in provision expense										0	0	0
Net Income (loss) from continuing operations										3,062	3,829	(149)
Operating Segments | Financial Holding Company
Segment Reporting Information [Line Items]
Net income (loss)										312	(3,058)	(1,979)
Increase (decrease) in provision expense										0	0	0
Net Income (loss) from continuing operations										(3,985)	(3,058)	(1,979)
Operating Segments | Insurance
Segment Reporting Information [Line Items]
Net income (loss)										(362)	81	(624)
Increase (decrease) in provision expense										0	0	0
Net Income (loss) from continuing operations										0	0	0
USI Insurance Services | Discontinued Operations, Disposed of by Sale
Segment Reporting Information [Line Items]
Gain on sale of subsidiary	$ 6,900									$ 6,926	$ 0	$ 0