FAIR VALUE MEASUREMENT - Carrying Values and Fair Values of Financial Assets Recorded at Fair Value on a Recurring Basis (Details) - USD ($) $ in Thousands	Jun. 30, 2020		Dec. 31, 2019		Jun. 30, 2019
Investments in:
Excess mortgage servicing rights assets, at fair value	$ 458,923		$ 505,343
Mortgage servicing rights, at fair value	5,021,086		5,686,233
Mortgage servicing rights financing receivables, at fair value	1,469,927		1,718,273
Real estate and other securities	6,144,236		19,477,728
Residential mortgage loans, held-for-sale	3,519,739		6,042,664
Residential mortgage loans subject to repurchase	1,075,008	[1]	172,336	[1]	$ 141,581
Derivative assets	166,613		41,501
Restricted cash	138,932		162,197
Liabilities
Residential mortgage loan repurchase liability	1,100,000
Derivative liabilities	24,936		6,885
Excess spread financing, at fair value	22,059		$ 31,777
Recurring Basis
Investments in:
Excess mortgage servicing rights, principal balance	81,666,625
Excess mortgage servicing rights, equity method investees, principal balance	31,923,937
Mortgage servicing rights, principal balance	379,397,792
Mortgage servicing rights financing receivables, principal balance	114,453,487
Servicer advance investments, principal balance	467,339
Real estate and other securities, available-for-sale, principal balance	27,043,911
Residential mortgage loans, held-for-sale, principal	853,934
Residential mortgage loans, held-for-sale, at fair value principal balance	2,921,615
Residential mortgage loans, held-for-investment, at fair value, principal balance	830,117
Residential mortgage loans, subject to repurchase, principal balance	1,075,008
Consumer loans, held-for-investment, principal balance	718,385
Derivative assets, principal balance	15,353,568
Note receivable, principal balance	46,724
Cash and cash equivalents, principal balance	1,013,208
Restricted cash, principal balance	138,932
Liabilities
Repurchase agreements, principal balance	9,179,077
Residential mortgage loan repurchase liability, principal balance	1,075,008
Derivative liabilities, principal amount	9,230,520
Excess spread financing, principal balance	2,655,595
Mortgage backed securities, issued at fair value	258,800
Recurring Basis | Notes Payable
Liabilities
Notes and bonds payable, principal balance	6,897,367
Recurring Basis | Term Loan
Liabilities
Notes and bonds payable, principal balance	533,383
Recurring Basis | Carrying Value
Investments in:
Excess mortgage servicing rights assets, at fair value	346,450
Excess mortgage servicing rights, equity method investees, at fair value	112,473
Mortgage servicing rights, at fair value	3,551,159
Mortgage servicing rights financing receivables, at fair value	1,469,927
Servicer advance investments, at fair value	559,011
Real estate and other securities	6,144,236
Residential mortgage loans, held-for-sale	694,830
Residential mortgage loans, held-for-sale, at fair value	2,824,909
Residential mortgage loans, held-for-investment, at fair value	750,332
Residential mortgage loans subject to repurchase	1,075,008
Consumer loans, held-for-investment, at fair value	766,462
Derivative assets	166,613
Note receivable	42,787
Cash and cash equivalents	1,013,208
Restricted cash	138,932
Other assets	42,769
Assets, fair value	19,699,106
Liabilities
Repurchase agreements	9,171,498
Residential mortgage loan repurchase liability	1,075,008
Derivative liabilities	24,936
Excess spread financing, at fair value	22,059
Contingent consideration	13,274
Liabilities, fair value	17,719,620
Recurring Basis | Carrying Value | Notes Payable
Liabilities
Notes and bonds payable	6,879,462
Recurring Basis | Carrying Value | Term Loan
Liabilities
Notes and bonds payable	533,383
Recurring Basis | Total
Investments in:
Excess mortgage servicing rights assets, at fair value	346,450
Excess mortgage servicing rights, equity method investees, at fair value	112,473
Mortgage servicing rights, at fair value	3,551,159
Mortgage servicing rights financing receivables, at fair value	1,469,927
Servicer advance investments, at fair value	559,011
Real estate and other securities	6,144,236
Residential mortgage loans, held-for-sale	701,610
Residential mortgage loans, held-for-sale, at fair value	2,824,909
Residential mortgage loans, held-for-investment, at fair value	750,332
Residential mortgage loans subject to repurchase	1,075,008
Consumer loans, held-for-investment, at fair value	766,462
Derivative assets	166,613
Note receivable	42,787
Cash and cash equivalents	1,013,208
Restricted cash	138,932
Other assets	42,769
Assets, fair value	19,705,886
Liabilities
Repurchase agreements	9,179,077
Residential mortgage loan repurchase liability	1,075,008
Derivative liabilities	24,936
Excess spread financing, at fair value	22,059
Contingent consideration	13,274
Liabilities, fair value	17,086,660
Recurring Basis | Total | Notes Payable
Liabilities
Notes and bonds payable	6,238,923
Recurring Basis | Total | Term Loan
Liabilities
Notes and bonds payable	533,383
Recurring Basis | Total | Level 1
Investments in:
Excess mortgage servicing rights assets, at fair value	0
Excess mortgage servicing rights, equity method investees, at fair value	0
Mortgage servicing rights, at fair value	0
Mortgage servicing rights financing receivables, at fair value	0
Servicer advance investments, at fair value	0
Real estate and other securities	0
Residential mortgage loans, held-for-sale	0
Residential mortgage loans, held-for-sale, at fair value	0
Residential mortgage loans, held-for-investment, at fair value	0
Residential mortgage loans subject to repurchase	0
Consumer loans, held-for-investment, at fair value	0
Derivative assets	0
Note receivable	0
Cash and cash equivalents	1,013,208
Restricted cash	138,932
Other assets	3,831
Assets, fair value	1,155,971
Liabilities
Repurchase agreements	0
Residential mortgage loan repurchase liability	0
Derivative liabilities	0
Excess spread financing, at fair value	0
Contingent consideration	0
Liabilities, fair value	0
Recurring Basis | Total | Level 1 | Notes Payable
Liabilities
Notes and bonds payable	0
Recurring Basis | Total | Level 1 | Term Loan
Liabilities
Notes and bonds payable	0
Recurring Basis | Total | Level 2
Investments in:
Excess mortgage servicing rights assets, at fair value	0
Excess mortgage servicing rights, equity method investees, at fair value	0
Mortgage servicing rights, at fair value	0
Mortgage servicing rights financing receivables, at fair value	0
Servicer advance investments, at fair value	0
Real estate and other securities	4,205,260
Residential mortgage loans, held-for-sale	0
Residential mortgage loans, held-for-sale, at fair value	1,719,899
Residential mortgage loans, held-for-investment, at fair value	0
Residential mortgage loans subject to repurchase	1,075,008
Consumer loans, held-for-investment, at fair value	0
Derivative assets	2,271
Note receivable	0
Cash and cash equivalents	0
Restricted cash	0
Other assets	0
Assets, fair value	7,002,438
Liabilities
Repurchase agreements	9,179,077
Residential mortgage loan repurchase liability	1,075,008
Derivative liabilities	24,539
Excess spread financing, at fair value	0
Contingent consideration	0
Liabilities, fair value	10,278,624
Recurring Basis | Total | Level 2 | Notes Payable
Liabilities
Notes and bonds payable	0
Recurring Basis | Total | Level 2 | Term Loan
Liabilities
Notes and bonds payable	0
Recurring Basis | Total | Level 3
Investments in:
Excess mortgage servicing rights assets, at fair value	346,450
Excess mortgage servicing rights, equity method investees, at fair value	112,473
Mortgage servicing rights, at fair value	3,551,159
Mortgage servicing rights financing receivables, at fair value	1,469,927
Servicer advance investments, at fair value	559,011
Real estate and other securities	1,938,976
Residential mortgage loans, held-for-sale	701,610
Residential mortgage loans, held-for-sale, at fair value	1,105,010
Residential mortgage loans, held-for-investment, at fair value	750,332
Residential mortgage loans subject to repurchase	0
Consumer loans, held-for-investment, at fair value	766,462
Derivative assets	164,342
Note receivable	42,787
Cash and cash equivalents	0
Restricted cash	0
Other assets	38,938
Assets, fair value	11,547,477
Liabilities
Repurchase agreements	0
Residential mortgage loan repurchase liability	0
Derivative liabilities	397
Excess spread financing, at fair value	22,059
Contingent consideration	13,274
Liabilities, fair value	6,808,036
Recurring Basis | Total | Level 3 | Notes Payable
Liabilities
Notes and bonds payable	6,238,923
Recurring Basis | Total | Level 3 | Term Loan
Liabilities
Notes and bonds payable	533,383
Recurring Basis | Total | Fair Value Measured at Net Asset Value Per Share
Investments in:
Other assets	$ 31,800

[1] See Note 8 for details.