CONSOLIDATED STATEMENTS OF OPERATIONS - USD ($)	12 Months Ended
	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Interest income
Loans, including fees	$ 127,106,000	$ 95,315,000	$ 79,081,000
Investment securities:
Taxable interest	42,286,000	41,993,000	36,121,000
Tax-exempt interest	170,000	207,000	306,000
Federal funds sold / securities purchased under agreements to resell		1,708,000	1,310,000
Interest earning deposits	10,007,000	8,737,000	5,202,000
Total interest income	179,569,000	147,960,000	122,020,000
Interest expense
Deposits	34,400,000	25,946,000	14,418,000
Short-term borrowings	3,131,000	451,000	336,000
Subordinated debentures	750,000	714,000	586,000
Total interest expense	38,281,000	27,111,000	15,340,000
Net interest income	141,288,000	120,849,000	106,680,000
Provision for loan and lease losses	4,400,000	3,585,000	2,920,000
Net interest income after provision for loan and lease losses	136,888,000	117,264,000	103,760,000
Non-interest income
Prepaid, debit card and related fees	65,141,000	54,627,000	53,367,000
Net realized and unrealized gains on commercial loans originated for sale	24,072,000	20,498,000	17,919,000
Gain on sale of investment securities		41,000	2,231,000
Gain on sale of IRA portfolio		65,000,000
Gain on sale of health savings portfolio			2,538,000
Change in value of investment in unconsolidated entity		(3,689,000)	(20,000)
Leasing related income	3,243,000	3,071,000	2,663,000
Affinity fees		281,000	1,545,000
Loss from sale of European prepaid card operations			(3,437,000)
Other	2,220,000	1,691,000	1,636,000
Total non-interest income	104,127,000	153,795,000	91,548,000
Non-interest expense
Salaries and employee benefits	94,259,000	79,816,000	75,832,000
Depreciation and amortization	3,696,000	3,997,000	4,452,000
Rent and related occupancy cost	6,628,000	5,474,000	5,680,000
Data processing expense	4,894,000	6,187,000	10,159,000
One-time fee to exit data processing contract			1,136,000
Printing and supplies	637,000	906,000	1,354,000
Audit expense	1,785,000	2,002,000	1,682,000
Legal expense	5,319,000	7,845,000	8,072,000
Amortization of intangible assets	1,531,000	1,531,000	1,528,000
FDIC Insurance	7,025,000	8,819,000	10,097,000
Software	12,731,000	13,304,000	12,597,000
Insurance	2,475,000	2,578,000	2,333,000
Telecom and IT network communications	1,493,000	1,373,000	1,793,000
Securitization and servicing expense	81,000	117,000	170,000
Consulting	3,240,000	3,239,000	2,227,000
Civil money penalties	8,900,000	(290,000)	2,290,000
Prepaid relationship exit expense		672,000
Lease termination expense	908,000	395,000
Other	12,919,000	13,313,000	13,512,000
Total non-interest expense	168,521,000	151,278,000	154,914,000
Income from continuing operations before income taxes	72,494,000	119,781,000	40,394,000
Income tax expense	21,226,000	32,241,000	23,056,000
Net income from continuing operations	51,268,000	87,540,000	17,338,000
Discontinued operations
Income from discontinued operations before income taxes	510,000	1,491,000	4,059,000
Income tax expense (benefit)	219,000	354,000	(276,000)
Income from discontinued operations, net of tax	291,000	1,137,000	4,335,000
Net income	$ 51,559,000	$ 88,677,000	$ 21,673,000
Net income per share from continuing operations - basic	$ 0.90	$ 1.55	$ 0.31
Net income per share from discontinued operations - basic	0.01	0.02	0.08
Net income per share - basic	0.91	1.57	0.39
Net income per share from continuing operations - diluted	0.89	1.53	0.31
Net income per share from discontinued operations - diluted	0.01	0.02	0.08
Net income per share - diluted	$ 0.90	$ 1.55	$ 0.39
Service Fees On Deposit Accounts [Member]
Non-interest income
Non-interest income	$ 75,000	$ 3,622,000	$ 6,788,000
ACH, Card And Other Payment Processing Fees [Member]
Non-interest income
Non-interest income	$ 9,376,000	$ 8,653,000	$ 6,318,000