Income Tax - Temporary Differences of Deferred Tax Assets/Liabilities (Details) - USD ($) $ in Thousands	Dec. 31, 2017	Dec. 31, 2016
Components of Deferred Tax Assets [Abstract]
Allowance for loan losses	$ 7,575	$ 11,571
Deferred loan fees, net	553	2,433
Deferred compensation	2,334	2,541
Other real estate	317	524
State tax carryforward	2,312	1,724
Stock benefits/compensation	1,032	1,580
FDIC acquired assets	306	331
Loan mark-to-market adjustment	479	1,088
Net operating losses	1,497	2,708
Other	660	1,179
Total	17,065	25,679
Components of Deferred Tax Liabilities [Abstract]
Mortgage servicing rights	21,566	26,857
Accelerated depreciation	1,637	2,721
Unrealized gains on investment securities available-for-sale	159	424
FDIC acquired assets		0
Deductible prepaids	333	214
Other	379	504
Total	24,074	30,720
Net deferred tax liability	$ (7,009)	$ (5,041)