Segment Information - Components of Corporate/Other (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2016	Sep. 30, 2016	Jun. 30, 2016	Mar. 31, 2016	Dec. 31, 2015	Sep. 30, 2015	Jun. 30, 2015	Mar. 31, 2015	Dec. 31, 2016	Dec. 31, 2015	Dec. 31, 2014
Segment Reporting Information [Line Items]
Interest income	$ 56,334	$ 60,906	$ 66,787	$ 62,328	$ 68,829	$ 63,484	$ 63,373	$ 63,746	$ 246,355	$ 259,432	$ 242,070
Interest expense	(20,537)	(21,597)	(22,444)	(23,950)	(25,039)	(23,875)	(23,008)	(23,961)	(88,528)	(95,883)	(87,463)
Net Interest Income	35,797	39,309	44,343	38,378	43,790	39,609	40,365	39,785	157,827	163,549	154,607
Reversal of provision for loan losses	0	859	6,532	(289)					7,102	355	(961)
Investment fair value changes, net									(28,574)	(21,357)	(10,202)
Realized gains, net									28,009	36,369	15,478
Total non-interest income, net	9,763	33,712	10,888	4,454	19,593	(3,412)	14,104	(5,031)	58,817	25,254	37,790
Direct operating expenses	(17,824)	(20,355)	(20,155)	(30,452)	(22,638)	(24,497)	(25,218)	(25,063)	(88,786)	(97,416)	(90,123)
(Provision for) benefit from income taxes									(3,708)	10,346	(744)
Net Income	$ 25,355	$ 52,553	$ 41,281	$ 12,063	$ 41,059	$ 19,164	$ 27,064	$ 14,801	131,252	102,088	100,569
Corporate/ Other
Segment Reporting Information [Line Items]
Interest income									19,740	24,844	25,798
Interest expense									(50,942)	(53,663)	(47,003)
Net Interest Income									(31,202)	(28,819)	(21,205)
Reversal of provision for loan losses									0	0	(877)
Investment fair value changes, net									(4,207)	(1,268)	(1,024)
Other income									0	0	1,600
Realized gains, net									292	0	1,701
Total non-interest income, net									(3,915)	(1,268)	2,277
Direct operating expenses									(53,761)	(38,557)	(37,454)
(Provision for) benefit from income taxes									0	3,878	(76)
Net Income									(88,878)	(64,766)	(57,335)
Corporate/ Other | Legacy Consolidated VIEs
Segment Reporting Information [Line Items]
Interest income									19,537	24,814	25,786
Interest expense									(13,103)	(15,646)	(20,844)
Net Interest Income									6,434	9,168	4,942
Reversal of provision for loan losses									0	0	(877)
Investment fair value changes, net									(4,200)	(1,192)	(894)
Other income									0	0	0
Realized gains, net									0	0	1,701
Total non-interest income, net									(4,200)	(1,192)	807
Direct operating expenses									0	0	(165)
(Provision for) benefit from income taxes									0	0	0
Net Income									2,234	7,976	4,707
Corporate/ Other | Other
Segment Reporting Information [Line Items]
Interest income									203	30	12
Interest expense									(37,839)	(38,017)	(26,159)
Net Interest Income									(37,636)	(37,987)	(26,147)
Reversal of provision for loan losses									0	0	0
Investment fair value changes, net									(7)	(76)	(130)
Other income									0	0	1,600
Realized gains, net									292	0	0
Total non-interest income, net									285	(76)	1,470
Direct operating expenses									(53,761)	(38,557)	(37,289)
(Provision for) benefit from income taxes									0	3,878	(76)
Net Income									$ (91,112)	$ (72,742)	$ (62,042)