Income Taxes - Components of deferred tax assets and liabilities (Details) - USD ($) $ in Thousands	Dec. 31, 2019	Dec. 31, 2018
Deferred tax assets:
Purchase accounting fair value adjustment	$ 10,441	$ 16,239
Statutory bad debt deduction less than financial statement provision	23,588	22,904
Net operating loss carry-forward	1,834	2,092
Investment security provision	474	593
State tax deductions	3,810	4,240
Accrued compensation	151	148
Deferred compensation	94	175
Mark to market on loans held for sale	913	260
Depreciation	788	202
Nonaccrual loan interest	5,550	6,027
Other real estate owned	533	585
Unrealized loss on securities available for sale	0	13,631
Non-qualified stock option and restricted share expense	2,348	1,420
Goodwill	31	117
Lease expense	0	60
Lease expense - ROU asset	614
Other	2,162	2,013
Total deferred tax assets	53,331	70,706
Deferred tax liabilities:
FHLB stock dividends	(408)	(617)
Deferred loan costs	(7,441)	(6,816)
State taxes deferred and other	(2,516)	(2,655)
Prepaid expenses	(1,359)	(1,802)
Amortization of intangibles	(3,837)	(4,524)
Unrealized gain on securities available for sale	(4,084)	0
Other	(2,023)	(3,379)
Total deferred tax liabilities	(21,668)	(19,793)
Net deferred tax assets	$ 31,663	$ 50,913