Income Taxes - Components of Deferred Tax Assets and Liabilities (Details) - USD ($) $ in Thousands	Dec. 31, 2019	Dec. 31, 2018
Deferred tax assets
Allowance for loan losses	$ 14,304	$ 14,097
Loans	10,284	18,655
Deferred compensation	12,050	10,001
Net unrealized losses on securities	0	6,180
Impairment of assets	1,108	1,280
Net operating loss carryforwards	9,387	19,065
Lease liabilities under operating leases	22,686	0
Other	934	9,800
Total deferred tax assets	70,753	79,078
Deferred tax liabilities
Net unrealized gains on securities	190	0
Investment in partnerships	967	1,572
Fixed assets	2,952	3,865
Mortgage servicing rights	13,472	12,350
Junior subordinated debt	2,304	1,607
Intangibles	0	6,190
Lease right-of-use asset	21,727	0
Other	1,859	1,792
Total deferred tax liabilities	43,471	27,376
Net deferred tax assets	$ 27,282	$ 51,702