Income Taxes - Components (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2016	Dec. 31, 2015	Dec. 31, 2014
Components of income tax expense (benefit)
Current federal	$ 399	$ 220	$ 122
Current state			6
Deferred federal	6,824	7,023	3,120
Deferred state	1,597	1,733	4,876
Change in valuation allowance			(2,363)
Provision for income taxes	8,820	8,976	$ 5,761
Deferred tax assets
Allowance for loan losses	7,057	7,099
Deferred compensation	735	690
Amortization of core deposit	1,401	1,629
Goodwill amortization/impairment	10,127	11,623
Stock option expense	946	814
OREO write downs	5,110	6,917
Federal net operating loss ("NOL") carryforward	19,362	24,105
State net operating loss ("NOL") carryforward	7,735	8,746
Deferred tax credit	2,058	1,749
Other assets	1,199	792
Total deferred tax assets	55,730	64,164
Deferred tax liabilities
Accumulated depreciation on premises and equipment	(681)	(601)
Mortgage servicing rights	(2,760)	(2,484)
State tax benefits	(4,127)	(4,686)
Other liabilities	(526)	(336)
Total deferred tax liabilities	(8,094)	(8,107)
Net deferred tax asset before valuation allowance	47,636	56,057
Tax effect of adjustments related to other comprehensive loss	5,828	8,495
Net deferred tax asset	$ 53,464	$ 64,552