Segment Financials (Operating Segments) (Details) - USD ($)	3 Months Ended								12 Months Ended
	Dec. 31, 2019	Sep. 30, 2019	Jun. 30, 2019	Mar. 31, 2019	Dec. 31, 2018	Sep. 30, 2018	Jun. 30, 2018	Mar. 31, 2018	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Segment Reporting Information [Line Items]
Interest income	$ 43,536,000	$ 48,375,000	$ 44,080,000	$ 43,578,000	$ 38,637,000	$ 38,656,000	$ 35,360,000	$ 35,307,000	$ 179,569,000	$ 147,960,000	$ 122,020,000
Interest expense									38,281,000	27,111,000	15,340,000
Net interest income	35,179,000	37,560,000	34,539,000	34,010,000	30,609,000	30,632,000	29,534,000	30,074,000	141,288,000	120,849,000	106,680,000
Provision for loan and lease losses	1,450,000	650,000	600,000	1,700,000	925,000	1,060,000	900,000	700,000	4,400,000	3,585,000	2,920,000
Non-interest income	20,498,000	33,515,000	19,749,000	30,365,000	16,690,000	90,970,000	17,040,000	29,095,000	104,127,000	153,795,000	91,548,000
Non-interest expense	47,722,000	42,051,000	39,519,000	39,229,000	37,620,000	37,299,000	37,310,000	39,049,000	168,521,000	151,278,000	154,914,000
Income before income tax	6,505,000	28,374,000	14,169,000	23,446,000	8,754,000	83,243,000	8,364,000	19,420,000	72,494,000	119,781,000	40,394,000
Income tax expense	3,641,000	7,975,000	3,575,000	6,035,000	2,691,000	21,942,000	2,209,000	5,399,000	21,226,000	32,241,000	23,056,000
Net income from continuing operations	2,864,000	20,399,000	10,594,000	17,411,000	6,063,000	61,301,000	6,155,000	14,021,000	51,268,000	87,540,000	17,338,000
Net income (loss) from discontinued operations, net of tax	(1,010,000)	26,000	756,000	519,000	1,056,000	(24,000)	(14,000)	119,000	291,000	1,137,000	4,335,000
Net income	1,854,000	$ 20,425,000	$ 11,350,000	$ 17,930,000	7,119,000	$ 61,277,000	$ 6,141,000	$ 14,140,000	51,559,000	88,677,000	21,673,000
Total assets	5,656,963,000				4,437,911,000				5,656,963,000	4,437,911,000
Total liabilities	5,172,466,000				4,031,135,000				5,172,466,000	4,031,135,000
Specialty Finance [Member]
Segment Reporting Information [Line Items]
Interest income									126,814,000	95,221,000	78,464,000
Interest expense									1,429,000	3,970,000	3,455,000
Net interest income									125,385,000	91,251,000	75,009,000
Provision for loan and lease losses									4,400,000	3,585,000	2,920,000
Non-interest income									29,140,000	89,187,000	27,952,000
Non-interest expense									63,884,000	57,952,000	56,472,000
Income before income tax									86,241,000	118,902,000	43,569,000
Net income from continuing operations									86,241,000	118,902,000	43,569,000
Net income									86,241,000	118,902,000	43,569,000
Total assets	3,008,304,000				2,181,499,000				3,008,304,000	2,181,499,000
Total liabilities	247,485,000				281,326,000				247,485,000	281,326,000
Payments [Member]
Segment Reporting Information [Line Items]
Interest allocation									52,755,000	52,739,000	43,556,000
Interest expense									28,971,000	21,293,000	10,475,000
Net interest income									23,784,000	31,446,000	33,081,000
Non-interest income									74,742,000	63,939,000	61,781,000
Non-interest expense									67,884,000	65,894,000	71,029,000
Income before income tax									30,642,000	29,492,000	23,833,000
Net income from continuing operations									30,642,000	29,492,000	23,833,000
Net income									30,642,000	29,492,000	23,833,000
Total assets	57,746,000				43,737,000				57,746,000	43,737,000
Total liabilities	4,030,921,000				3,545,877,000				4,030,921,000	3,545,877,000
Corporate [Member]
Segment Reporting Information [Line Items]
Interest income									52,755,000	52,739,000	43,556,000
Interest allocation									(52,755,000)	(52,739,000)	(43,556,000)
Interest expense									7,881,000	1,849,000	1,410,000
Net interest income									(7,881,000)	(1,849,000)	(1,410,000)
Non-interest income									245,000	668,000	1,815,000
Non-interest expense									36,753,000	27,432,000	27,413,000
Income before income tax									(44,389,000)	(28,613,000)	(27,008,000)
Income tax expense									21,226,000	32,241,000	23,056,000
Net income from continuing operations									(65,615,000)	(60,854,000)	(50,064,000)
Net income									(65,615,000)	(60,854,000)	(50,064,000)
Total assets	2,450,256,000				2,014,844,000				2,450,256,000	2,014,844,000
Total liabilities	894,060,000				203,932,000				894,060,000	203,932,000
Discontinued Operations [Member]
Segment Reporting Information [Line Items]
Net income (loss) from discontinued operations, net of tax									291,000	1,137,000	4,335,000
Net income									291,000	1,137,000	$ 4,335,000
Total assets	$ 140,657,000				$ 197,831,000				$ 140,657,000	$ 197,831,000