Segment Financials (Operating Segments) (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2015	Sep. 30, 2015	Jun. 30, 2015	Mar. 31, 2015	Dec. 31, 2014	Sep. 30, 2014	Jun. 30, 2014	Mar. 31, 2014	Dec. 31, 2015	Dec. 31, 2014	Dec. 31, 2013
Segment Reporting Information [Line Items]
Interest income	$ 22,248	$ 21,193	$ 20,372	$ 19,717	$ 18,522	$ 18,033	$ 17,610	$ 16,555	$ 83,530	$ 70,720	$ 51,150
Interest expense									13,599	11,295	10,768
Net interest income	18,582	17,798	17,037	16,514	15,715	15,268	14,791	13,651	69,931	59,425	40,382
Provision for loan and lease losses	300	625	510	665	(1,404)	158	1,173	1,275	2,100	1,202	355
Non-interest income	70,267	17,299	24,724	20,777	17,675	20,307	23,389	23,678	133,067	85,049	82,073
Non-interest expense	58,999	47,795	46,434	40,860	37,628	33,135	34,012	31,205	194,088	135,980	101,817
Income before income tax	29,550	(13,323)	(5,183)	(4,234)	(2,834)	2,282	2,995	4,849	6,810	7,292	20,283
Income taxes	12,267	(5,706)	(2,684)	(2,427)	(13,929)	(3,560)	1,343	1,623	1,450	(14,523)	6,767
Net income from continuing operations	17,283	(7,617)	(2,499)	(1,807)	11,095	5,842	1,652	3,226	5,360	21,815	13,516
Net income (loss) from discontinued operations, net of tax	1,336	2,042	2,673	2,021	9,096	19,127	8,670	(1,599)	8,072	35,294	(27,938)
Net income (loss) available to common shareholders	18,619	$ (5,575)	$ 174	$ 214	20,191	$ 24,969	$ 10,322	$ 1,627	13,432	57,109	(14,422)
Total assets	4,765,823				4,986,317				4,765,823	4,986,317
Total liabilities	4,445,822				4,667,294				4,445,822	4,667,294
Specialty Finance [Member]
Segment Reporting Information [Line Items]
Interest income									49,789	36,402	27,577
Interest expense									5,039	3,964	3,762
Net interest income									44,750	32,438	23,815
Provision for loan and lease losses									2,100	1,202	355
Non-interest income									17,891	19,030	23,049
Non-interest expense									47,863	37,933	30,190
Income before income tax									12,678	12,333	16,319
Net income from continuing operations									12,678	12,333	16,319
Net income (loss) available to common shareholders									12,678	12,333	16,319
Total assets	1,747,558				1,099,914				1,747,558	1,099,914
Total liabilities	783,866				1,165,567				783,866	1,165,567
Payments [Member]
Segment Reporting Information [Line Items]
Interest income									11	58	64
Interest allocation									33,730	34,261	23,510
Interest expense									7,445	6,137	4,796
Net interest income									26,296	28,182	18,778
Non-interest income									97,963	65,509	57,129
Non-interest expense									128,828	79,145	59,202
Income before income tax									(4,569)	14,546	16,705
Net income from continuing operations									(4,569)	14,546	16,705
Net income (loss) available to common shareholders									(4,569)	14,546	16,705
Total assets	35,165				30,503				35,165	30,503
Total liabilities	2,991,856				3,198,129				2,991,856	3,198,129
Corporate [Member]
Segment Reporting Information [Line Items]
Interest income									33,730	34,260	23,509
Interest allocation									(33,730)	(34,261)	(23,510)
Interest expense									1,115	1,194	2,210
Net interest income									(1,115)	(1,195)	(2,211)
Non-interest income									17,213	510	1,895
Non-interest expense									17,397	18,902	12,425
Income before income tax									(1,299)	(19,587)	(12,741)
Income taxes									1,450	(14,523)	6,767
Net income from continuing operations									(2,749)	(5,064)	(19,508)
Net income (loss) available to common shareholders									(2,749)	(5,064)	(19,508)
Total assets	2,399,191				2,967,971				2,399,191	2,967,971
Total liabilities	670,100				303,598				670,100	303,598
Discontinued Operations [Member]
Segment Reporting Information [Line Items]
Net income (loss) from discontinued operations, net of tax									8,072	35,294	(27,938)
Net income (loss) available to common shareholders									8,072	35,294	$ (27,938)
Total assets	$ 583,909				$ 887,929				$ 583,909	$ 887,929