Reportable Segments Reconciliation of Reportable Segment Items (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2020	Mar. 31, 2019	Dec. 31, 2019
Segment Reporting Information [Line Items]
Net interest income	$ 57,257	$ 58,338
Fees and other income	21,521	25,248
Total revenue	78,778	83,586
Provision/(credit) for loan losses	16,962	(1,426)
Operating expense	60,908	60,553
Income before income taxes	908	24,459
Income tax expense	102	4,917
Net income before attribution to noncontrolling interests	806	19,542
Noncontrolling interests	6	100
Net income attributable to the Company	800	19,442
Assets	8,746,326	8,572,143	$ 8,830,501
Amortization of intangibles	715	672
Depreciation	2,959	3,084
Restructuring charges	0	1,646
Operating Segments | Private Banking (1)
Segment Reporting Information [Line Items]
Net interest income	58,090	59,337
Fees and other income	1,108	3,258
Total revenue	59,198	62,595
Provision/(credit) for loan losses	16,962	(1,426)
Operating expense	42,588	41,317
Income before income taxes	(352)	22,704
Income tax expense	(931)	4,430
Net income before attribution to noncontrolling interests	579	18,274
Net income attributable to the Company	579	18,274
Assets	8,692,069	8,481,160
Amortization of intangibles	77	0
Depreciation	2,626	2,670
Operating Segments | Wealth Management and Trust (1)
Segment Reporting Information [Line Items]
Net interest income	72	106
Fees and other income	18,485	19,126
Total revenue	18,557	19,232
Operating expense	15,449	15,567
Income before income taxes	3,108	3,665
Income tax expense	1,074	1,194
Net income before attribution to noncontrolling interests	2,034	2,471
Net income attributable to the Company	2,034	2,471
Assets	143,998	147,771
Amortization of intangibles	638	672
Depreciation	294	367
Holding Company and Eliminations
Segment Reporting Information [Line Items]
Net interest income	(905)	(1,105)
Fees and other income	1,928	2,864
Total revenue	1,023	1,759
Operating expense	2,871	3,669
Income before income taxes	(1,848)	(1,910)
Income tax expense	(41)	(707)
Net income before attribution to noncontrolling interests	(1,807)	(1,203)
Noncontrolling interests	6	100
Net income attributable to the Company	(1,813)	(1,303)
Assets	(89,741)	(56,788)
Depreciation	$ 39	47
Interest expense | Operating Segments | Private Banking (1)
Segment Reporting Information [Line Items]
Restructuring charges		1,300
Interest expense | Operating Segments | Wealth Management and Trust (1)
Segment Reporting Information [Line Items]
Restructuring charges		$ 400