CONSOLIDATED VARIABLE INTEREST ENTITIES - Variable Interest Entities, Assets and Liabilities (Details) - USD ($) $ in Thousands		Jun. 30, 2020	Dec. 31, 2019
Assets
Total	[1]	$ 559,011	$ 581,777
Cash and cash equivalents	[1]	1,013,208	528,737
Restricted cash		138,932	162,197
Other assets		1,194,057	1,395,800
Total assets		23,752,149	44,863,454
Liabilities
Notes and bonds payable	[1]	6,879,462	7,720,148
Accrued expenses and other liabilities	[1]	532,249	600,790
Total liabilities		18,363,177	37,627,194
VIE, consolidated
Assets
Total		544,204	565,271
Residential mortgage loans, held-for-investment, at fair value		417,560	913,030
Consumer loans, held-for-investment, at fair value		761,525	818,943
Cash and cash equivalents		65,661	53,867
Restricted cash		13,896	14,423
Other assets		19,666	18,379
Total assets		1,822,512	2,383,913
Liabilities
Notes and bonds payable		1,413,132	1,913,696
Accrued expenses and other liabilities		12,341	20,038
Total liabilities		1,425,473	1,933,734
VIE, consolidated | The Buyer
Assets
Total		544,204	565,271
Residential mortgage loans, held-for-investment, at fair value		0	0
Consumer loans, held-for-investment, at fair value		0	0
Cash and cash equivalents		36,878	30,065
Restricted cash		5,331	5,350
Other assets		6	2,414
Total assets		586,419	603,100
Liabilities
Notes and bonds payable		433,635	433,300
Accrued expenses and other liabilities		2,104	1,593
Total liabilities		435,739	434,893
VIE, consolidated | Shelter Joint Ventures
Assets
Total		0	0
Residential mortgage loans, held-for-investment, at fair value		0	0
Consumer loans, held-for-investment, at fair value		0	0
Cash and cash equivalents		28,783	23,802
Restricted cash		0	0
Other assets		8,501	3,556
Total assets		37,284	27,358
Liabilities
Notes and bonds payable		0	0
Accrued expenses and other liabilities		6,229	4,187
Total liabilities		6,229	4,187
VIE, consolidated | Residential Mortgage Loans
Assets
Total		0	0
Residential mortgage loans, held-for-investment, at fair value		417,560	913,030
Consumer loans, held-for-investment, at fair value		0	0
Cash and cash equivalents		0	0
Restricted cash		0	0
Other assets		350	0
Total assets		417,910	913,030
Liabilities
Notes and bonds payable		258,807	659,738
Accrued expenses and other liabilities		0	10,132
Total liabilities		258,807	669,870
VIE, consolidated | Consumer Loan Companies
Assets
Total		0	0
Residential mortgage loans, held-for-investment, at fair value		0	0
Consumer loans, held-for-investment, at fair value		761,525	818,943
Cash and cash equivalents		0	0
Restricted cash		8,565	9,073
Other assets		10,809	12,409
Total assets		780,899	840,425
Liabilities
Notes and bonds payable		720,690	820,658
Accrued expenses and other liabilities		4,008	4,126
Total liabilities		$ 724,698	$ 824,784

[1] See Note 13 regarding consolidated VIEs.