Segment Financials (Operating Segments) (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2020	Mar. 31, 2019	Dec. 31, 2019
Segment Reporting Information [Line Items]
Interest income	$ 51,466	$ 43,578
Interest expense	8,555	9,568
Net interest income	42,911	34,010
Provision for credit losses	3,579	1,700
Non-interest income	16,599	30,365
Non-interest expense	38,418	39,229
Income before income tax	17,513	23,446
Income tax expense	4,352	6,035
Net income from continuing operations	13,161	17,411
Income (Loss) from discontinued operations	(570)	519
Net income	12,591	17,930
Total assets	5,458,108		$ 5,656,963
Total liabilities	4,960,004		5,172,466
Specialty Finance [Member]
Segment Reporting Information [Line Items]
Interest income	39,293	30,473
Interest expense	340	368
Net interest income	38,953	30,105
Provision for credit losses	3,579	1,700
Non-interest income	(3,839)	11,777
Non-interest expense	16,916	15,357
Income before income tax	14,619	24,825
Net income from continuing operations	14,619	24,825
Net income	14,619	24,825
Total assets	3,735,854		3,008,304
Total liabilities	260,220		247,485
Payments [Member]
Segment Reporting Information [Line Items]
Interest allocation	12,173	13,105
Interest expense	5,065	8,384
Net interest income	7,108	4,721
Non-interest income	20,421	18,548
Non-interest expense	17,145	16,459
Income before income tax	10,384	6,810
Net income from continuing operations	10,384	6,810
Net income	10,384	6,810
Total assets	48,361		57,746
Total liabilities	3,977,947		4,030,921
Corporate [Member]
Segment Reporting Information [Line Items]
Interest income	12,173	13,105
Interest allocation	(12,173)	(13,105)
Interest expense	3,150	816
Net interest income	(3,150)	(816)
Non-interest income	17	40
Non-interest expense	4,357	7,413
Income before income tax	(7,490)	(8,189)
Income tax expense	4,352	6,035
Net income from continuing operations	(11,842)	(14,224)
Net income	(11,842)	(14,224)
Total assets	1,539,775		2,450,256
Total liabilities	721,837		894,060
Discontinued Operations [Member]
Segment Reporting Information [Line Items]
Income (Loss) from discontinued operations	(570)	519
Net income	(570)	$ 519
Total assets	$ 134,118		$ 140,657