FAIR VALUE MEASUREMENT - Carrying Values and Fair Values of Financial Assets and Liabilities Recorded at Fair Value on a Recurring Basis (Details) - USD ($) $ in Thousands	Dec. 31, 2018	Dec. 31, 2017	Dec. 31, 2016
Investments in:
Excess mortgage servicing rights, at fair value	$ 447,860	$ 1,173,713
Mortgage servicing rights financing receivables, at fair value	1,644,504	598,728
Real estate and other securities, available-for-sale	11,636,581	8,071,140
Residential mortgage loans, held for sale, at fair value	2,808,529	0
Residential mortgage loans, held-for-investment, fair value	121,088	0
Residential mortgage loans subject to repurchase, at fair value	121,602	0	$ 0
Derivative assets, at fair value	10,893	2,423
Restricted cash, at fair value	164,020	150,252
Liabilities:
Residential mortgage loan repurchase liability, at fair value	121,602	0
Derivative liabilities, at fair value	29,389	697
Excess spread financing, at fair value	39,304	0
Equity investment	74,323	0
Recurring Basis
Investments in:
Excess mortgage servicing rights, principal balance	106,426,363	217,121,299
Excess mortgage servicing rights, equity method investees, principal balance	41,707,963	50,501,054
Mortgage servicing rights, principal balance	258,462,703	172,454,150
Mortgage servicing rights, financing receivables, principal balance	130,516,565	64,344,893
Servicer advance investments, principal balance	620,050	3,581,876
Real estate and other securities, available-for-sale, principal balance	22,152,845	14,822,986
Residential mortgage loans, held-for-investment, principal balance	706,111	806,635
Residential mortgage loans, held-for-sale, principal balance	1,043,550	1,907,052
Residential mortgage loans, held for sale, principal balance	2,934,727
Residential mortgage loans, held-for-investment, principal balance	122,260
Residential mortgage loans subject to repurchase, principal balance	121,602
Consumer loans, principal balance	1,072,577	1,377,792
Derivative assets, principal balance	840,179	772,500
Cash and cash equivalents, principal balance	251,058	295,798
Restricted cash, principal balance	164,020	150,252
Other assets, principal balance		1,788,354
Liabilities:
Repurchase agreements, principal balance	15,555,156	8,663,747
Notes and bonds payable, principal balance	7,117,909	7,097,223
Residential mortgage loan repurchase liability, principal balance	121,602
Derivative liabilities, principal balance	15,759,782	4,115,100
Excess spread financing, principal balance	3,492,587
Recurring Basis | Carrying Value
Investments in:
Excess mortgage servicing rights, at fair value	447,860	1,173,713
Excess mortgage servicing rights, equity method investees, at fair value	147,964	171,765
Mortgage servicing rights, at fair value	2,884,100	1,735,504
Mortgage servicing rights financing receivables, at fair value	1,644,504	598,728
Servicer advance investments, at fair value	735,846	4,027,379
Real estate and other securities, available-for-sale	11,636,581	8,071,140
Residential mortgage loans, held-for-investment, at fair value	614,241	691,155
Residential mortgage loans, held-for-sale, at fair value	932,480	1,725,534
Residential mortgage loans, held for sale, at fair value	2,808,529
Residential mortgage loans, held-for-investment, fair value	121,088
Residential mortgage loans subject to repurchase, at fair value	121,602
Consumer loans, at fair value	1,072,202	1,374,263
Derivative assets, at fair value	10,893	2,423
Cash and cash equivalents, at fair value	251,058	295,798
Restricted cash, at fair value	164,020	150,252
Other assets, at fair value	16,991	28,802
Assets, fair value	23,609,959	20,046,456
Liabilities:
Repurchase agreements, at fair value	15,553,969	8,662,139
Notes and bonds payable, at fair value	7,102,266	7,084,391
Residential mortgage loan repurchase liability, at fair value	121,602
Derivative liabilities, at fair value	29,389	697
Excess spread financing, at fair value	39,304
Contingent consideration, at fair value	40,842
Liabilities, fair value	22,887,372	15,747,227
Recurring Basis | Fair Value
Investments in:
Excess mortgage servicing rights, at fair value	447,860	1,173,713
Excess mortgage servicing rights, equity method investees, at fair value	147,964	171,765
Mortgage servicing rights, at fair value	2,884,100	1,735,504
Mortgage servicing rights financing receivables, at fair value	1,644,504	598,728
Servicer advance investments, at fair value	735,846	4,027,379
Real estate and other securities, available-for-sale	11,636,581	8,071,140
Residential mortgage loans, held-for-investment, at fair value	625,321	694,692
Residential mortgage loans, held-for-sale, at fair value	958,970	1,794,210
Residential mortgage loans, held for sale, at fair value	2,808,529
Residential mortgage loans, held-for-investment, fair value	121,088
Residential mortgage loans subject to repurchase, at fair value	121,602
Consumer loans, at fair value	1,054,820	1,379,746
Derivative assets, at fair value	10,893	2,423
Cash and cash equivalents, at fair value	251,058	295,798
Restricted cash, at fair value	164,020	150,252
Other assets, at fair value	16,991	28,802
Assets, fair value	23,630,147	20,124,152
Liabilities:
Repurchase agreements, at fair value	15,555,156	8,663,747
Notes and bonds payable, at fair value	7,076,400	7,109,803
Residential mortgage loan repurchase liability, at fair value	121,602
Derivative liabilities, at fair value	29,389	697
Excess spread financing, at fair value	39,304
Contingent consideration, at fair value	40,842
Liabilities, fair value	22,862,693	15,774,247
Equity investment	74,300
Recurring Basis | Fair Value | Level 1
Investments in:
Excess mortgage servicing rights, at fair value	0	0
Excess mortgage servicing rights, equity method investees, at fair value	0	0
Mortgage servicing rights, at fair value	0	0
Mortgage servicing rights financing receivables, at fair value	0	0
Servicer advance investments, at fair value	0	0
Real estate and other securities, available-for-sale	0	0
Residential mortgage loans, held-for-investment, at fair value	0	0
Residential mortgage loans, held-for-sale, at fair value	0	0
Residential mortgage loans, held for sale, at fair value	0
Residential mortgage loans, held-for-investment, fair value	0
Residential mortgage loans subject to repurchase, at fair value	0
Consumer loans, at fair value	0	0
Derivative assets, at fair value	0	0
Cash and cash equivalents, at fair value	251,058	295,798
Restricted cash, at fair value	164,020	150,252
Other assets, at fair value	7,778	19,259
Assets, fair value	422,856	465,309
Liabilities:
Repurchase agreements, at fair value	0	0
Notes and bonds payable, at fair value	0	0
Residential mortgage loan repurchase liability, at fair value	0
Derivative liabilities, at fair value	0	0
Excess spread financing, at fair value	0
Contingent consideration, at fair value	0
Liabilities, fair value	0	0
Recurring Basis | Fair Value | Level 2
Investments in:
Excess mortgage servicing rights, at fair value	0	0
Excess mortgage servicing rights, equity method investees, at fair value	0	0
Mortgage servicing rights, at fair value	0	0
Mortgage servicing rights financing receivables, at fair value	0	0
Servicer advance investments, at fair value	0	0
Real estate and other securities, available-for-sale	2,665,618	2,096,351
Residential mortgage loans, held-for-investment, at fair value	0	0
Residential mortgage loans, held-for-sale, at fair value	0	0
Residential mortgage loans, held for sale, at fair value	213,882
Residential mortgage loans, held-for-investment, fair value	0
Residential mortgage loans subject to repurchase, at fair value	121,602
Consumer loans, at fair value	0	0
Derivative assets, at fair value	42	2,423
Cash and cash equivalents, at fair value	0	0
Restricted cash, at fair value	0	0
Other assets, at fair value	0	0
Assets, fair value	3,001,144	2,098,774
Liabilities:
Repurchase agreements, at fair value	15,555,156	8,663,747
Notes and bonds payable, at fair value	0	0
Residential mortgage loan repurchase liability, at fair value	121,602
Derivative liabilities, at fair value	29,166	697
Excess spread financing, at fair value	0
Contingent consideration, at fair value	0
Liabilities, fair value	15,705,924	8,664,444
Recurring Basis | Fair Value | Level 3
Investments in:
Excess mortgage servicing rights, at fair value	447,860	1,173,713
Excess mortgage servicing rights, equity method investees, at fair value	147,964	171,765
Mortgage servicing rights, at fair value	2,884,100	1,735,504
Mortgage servicing rights financing receivables, at fair value	1,644,504	598,728
Servicer advance investments, at fair value	735,846	4,027,379
Real estate and other securities, available-for-sale	8,970,963	5,974,789
Residential mortgage loans, held-for-investment, at fair value	625,321	694,692
Residential mortgage loans, held-for-sale, at fair value	958,970	1,794,210
Residential mortgage loans, held for sale, at fair value	2,594,647
Residential mortgage loans, held-for-investment, fair value	121,088
Residential mortgage loans subject to repurchase, at fair value	0
Consumer loans, at fair value	1,054,820	1,379,746
Derivative assets, at fair value	10,851	0
Cash and cash equivalents, at fair value	0	0
Restricted cash, at fair value	0	0
Other assets, at fair value	9,213	9,543
Assets, fair value	20,206,147	17,560,069
Liabilities:
Repurchase agreements, at fair value	0	0
Notes and bonds payable, at fair value	7,076,400	7,109,803
Residential mortgage loan repurchase liability, at fair value	0
Derivative liabilities, at fair value	223	0
Excess spread financing, at fair value	39,304
Contingent consideration, at fair value	40,842
Liabilities, fair value	7,156,769	$ 7,109,803
Equal to Greater than 90 Days Past Due | Recurring Basis | Fair Value
Investments in:
Residential mortgage loans, held for sale, at fair value	88,700
Residential mortgage loans, held-for-investment, fair value	400
Residential Mortgage Backed Securities | Notes and bonds payable
Liabilities:
Notes and bonds payable, at fair value	$ 117,000