Dispositions, Assets Held for Sale and Discontinued Operations - Assets and Liabilities Held for Sale and Revenue and Expenses (Details) - USD ($) $ in Thousands	3 Months Ended				12 Months Ended
	Dec. 31, 2018	Sep. 30, 2018	Jun. 30, 2018	Mar. 31, 2018	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017	Jan. 01, 2019
Disposal Group, Including Discontinued Operation, Assets [Abstract]
Loans, at fair value	$ 215,383				$ 108,894	$ 215,383
Other Investments	75,002				137,472	75,002
Total investments	696,927				736,936	696,927
Cash and cash equivalents	2,860				7,137	2,860	$ 10,533
Other assets	46,034				68,510	46,034
Assets held for sale	68,231				107,835	68,231
Disposal Group, Including Discontinued Operation, Liabilities [Abstract]
Debt, net	354,083				374,454	354,083
Liabilities held for sale	62,980				102,430	62,980
Operating Lease, Right-of-Use Asset	0				23,832	0		$ 32,052
Operating lease liability	0				29,491	0		$ 33,558
Disposal Group, Including Discontinued Operation, Balance Sheet Disclosures [Abstract]
Impairments on assets held for sale or discontinued operations					0	0
Revenues:
Other revenue					59,888	38,062	42,275
Expenses:
Employee compensation and benefits					129,479	113,557	115,949
Other expenses					99,744	77,901	74,439
Net income (loss) before taxes from discontinued operations	0	$ 0	$ 0	$ 624	0	624	(6,222)
Gain on sale of discontinued operations	10,676	0	0	46,184	0	56,860	0
Less: Provision (benefit) for income taxes	1,387	0	0	12,327	0	13,714	(2,224)
Net income (loss) from discontinued operations	9,289	$ 0	$ 0	$ 34,481	0	43,770	(3,998)
Held for sale | Luxury Disposition
Disposal Group, Including Discontinued Operation, Assets [Abstract]
Loans, at fair value	63,340				98,272	63,340
Other Investments	798				1,019	798
Total investments	64,138				99,291	64,138
Cash and cash equivalents	2,860				7,137	2,860
Notes and accounts receivable, net	230				238	230
Other assets	1,003				1,169	1,003
Assets held for sale	68,231				107,835	68,231
Disposal Group, Including Discontinued Operation, Liabilities [Abstract]
Debt, net	61,381				97,822	61,381
Other liabilities and accrued expenses (2)	1,599				4,608	1,599
Liabilities held for sale	$ 62,980				102,430	62,980
Operating Lease, Right-of-Use Asset					318
Operating lease liability					341
Discontinued Operations
Revenues:
Rental and related revenue					0	6,476	74,386
Other revenue					0	149	1,583
Total revenues					0	6,625	75,969
Expenses:
Employee compensation and benefits					0	2,788	30,215
Interest expense					0	1,252	13,068
Disposal Group, Including Discontinued Operation, Depreciation and Amortization					0	0	15,645
Other expenses					0	1,961	23,263
Total expenses					0	6,001	82,191
Net income (loss) before taxes from discontinued operations					0	624	(6,222)
Gain on sale of discontinued operations						56,860	0
Less: Provision (benefit) for income taxes					0	13,714	(2,224)
Net income (loss) from discontinued operations					0	$ 43,770	$ (3,998)
Warehouse Agreement Borrowings | Residential mortgage warehouse borrowings | Luxury Disposition
Disposal Group, Including Discontinued Operation, Liabilities [Abstract]
Debt instrument, maximum borrowing capacity					$ 154,500