Summary of Significant Accounting Policies (Narrative) (Details)			1 Months Ended	3 Months Ended	12 Months Ended
	Jul. 10, 2018 USD ($)	Nov. 29, 2012 USD ($)	May 31, 2016 USD ($)	Jun. 30, 2017 USD ($)	Dec. 31, 2019 USD ($) $ / shares loan shares	Dec. 31, 2018 USD ($) $ / shares shares	Dec. 31, 2017 USD ($) $ / shares shares	Dec. 31, 2011 $ / shares shares	Jan. 01, 2019 USD ($)	Sep. 30, 2014 USD ($)
Accounting Policies [Line Items]
Other than temporary impairment charges					$ 0	$ 0	$ 0
Book value of loans										$ 1,100,000,000
Amortization of intangible assets					1,531,000	1,531,000	1,528,000
Other real estate owned					0	0
Advertising costs					$ 782,000	$ 423,000	$ 435,000
Stock options included in dilutive earnings per share, due to exercise price per share being less than average market price | shares					971,604	1,160,000
Common stock options (in shares) | shares					340,000		1,152,625
Minimum exercisable prices (in dollars per share) | $ / shares					$ 6.75	$ 6.75	$ 7.36
Maximum exercisable prices (in dollars per share) | $ / shares					$ 9.58	$ 9.84	$ 10.45
Restricted cash and cash equivalents					$ 314,700,000	$ 294,200,000
Gross Carrying Amount					17,228,000	17,228,000
Amortization expense per year					499,000
Accumulated Amortization					$ 14,913,000	$ 13,382,000
Number of stock repurchased under, Stock repurchase program (in shares) | shares					100,000	100,000		100,000
Maximum number of shares authorized (in shares) | shares								750,000
Share repurchased during period, shares | shares					0	0	0	100,000
Gain on sale of health savings portfolio				$ 2,500,000			$ 2,538,000
Gain (Loss) on Disposition of Business							3,437,000
Number Of Loans Sold | loan					2
Average cost of repurchased stock (in dollars per share) | $ / shares								$ 8.66
Europe [Member]
Accounting Policies [Line Items]
Gain (Loss) on Disposition of Business				$ 3,400,000
Safe Harbor Individual Retirement Account Shira Portfolio [Member] | Discontinued Operations, Disposed of by Sale [Member]
Accounting Policies [Line Items]
Deposits, Savings Deposits	$ 400,000,000
Disposal Group, Including Discontinued Operation, Consideration	65,000,000
Accounting Standards Update 2018-02 [Member]
Accounting Policies [Line Items]
Reclassification from AOCI to retained earnings							812,000
Accounting Standards Update 2016-02 [Member]
Accounting Policies [Line Items]
Operating lease right-of-use									$ 16,400,000
Operating lease obligation									18,100,000
Accounting Standards Update 2016-02 [Member] | Restatement Adjustments [Member]
Accounting Policies [Line Items]
Accrued Rent									$ (1,700,000)
Maximum [Member] | Accounting Standards Update 2016-13 [Member]
Accounting Policies [Line Items]
New Accounting Pronouncement or Change in Accounting Principle, Effect of Adoption, Quantification					$ 3,000,000
Minimum [Member] | Accounting Standards Update 2016-13 [Member]
Accounting Policies [Line Items]
New Accounting Pronouncement or Change in Accounting Principle, Effect of Adoption, Quantification					$ 1,000,000
Internal Use Software [Member]
Accounting Policies [Line Items]
Estimated useful life					7 years
Total capitalized software costs					$ 7,500,000	$ 6,200,000
Amortization of intangible assets					2,300,000	2,400,000	$ 2,600,000
Gross Carrying Amount					1,800,000
Accumulated Amortization					$ 1,700,000	1,500,000
Software Intangible [Member]
Accounting Policies [Line Items]
Estimated useful life					8 years
Acquired finite-lived intangible assets		$ 1,800,000
Gross Carrying Amount					$ 1,817,000	1,817,000
Amortization expense per year					217,000
Amortization expense over remainder of amortization period					(159,000)
Accumulated Amortization					$ 1,658,000	1,468,000
Customer Lists, From Purchased Prepaid Cards [Member]
Accounting Policies [Line Items]
Estimated useful life					minus 12 years
Acquired finite-lived intangible assets					$ 12,000,000
Gross Carrying Amount					12,000,000
Amortization expense per year					1,000,000
Amortization expense over remainder of amortization period					0
Accumulated Amortization					12,000,000	11,000,000
Customer Lists, From Purchased Lease Receivables [Member]
Accounting Policies [Line Items]
Estimated useful life			10 years
Acquired finite-lived intangible assets			$ 3,400,000
Gross Carrying Amount					3,400,000	3,400,000
Amortization expense per year					340,000
Amortization expense over next five years					1,700,000
Accumulated Amortization					1,200,000	908,000
Purchase of lease receivables			$ 60,000,000
Fair Value, Measurements, Recurring [Member]
Accounting Policies [Line Items]
Loans held for sale					$ 1,180,546,000	$ 688,471,000
SHIRA Portfolio, Deposit Account [Member]
Accounting Policies [Line Items]
Non-interest income	$ 3,400,000