INVESTMENTS IN RESIDENTIAL MORTGAGE LOANS - Residential Mortgage Loans Outstanding by Loan Type, Excluding REO (Details) $ in Thousands	6 Months Ended
	Jun. 30, 2020 USD ($) loan	Dec. 31, 2019 USD ($)
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	$ 1,009,696
Carrying Value	$ 841,919
Weighted Average Life (Years)	1 year 2 months 12 days
Residential Mortgage Loans, Held-for-Investment, at Fair Value
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	$ 830,117
Carrying Value	$ 750,332	$ 925,706
Loan Count | loan	13,168
Weighted Average Yield	7.20%
Weighted Average Life (Years)	6 years 4 months 24 days
Acquired Residential Mortgage Loans Held-for-Sale
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	$ 853,934
Carrying Value	$ 694,830	1,429,052
Loan Count | loan	9,412
Weighted Average Yield	7.00%
Weighted Average Life (Years)	3 years 6 months
Acquired Reverse Mortgage Loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	$ 12,604
Carrying Value	$ 6,458	5,844
Loan Count | loan	29
Weighted Average Yield	7.90%
Weighted Average Life (Years)	3 years 10 months 24 days
Interest in reverse mortgage loans	70.00%
Unpaid principal balance	$ 600
Percentage of loans that have reached a termination event	51.00%
Acquired Reverse Mortgage Loans | Mr. Cooper
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest in reverse mortgage loans	30.00%
Acquired Performing Loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	$ 224,268
Carrying Value	$ 198,150	857,821
Loan Count | loan	4,680
Weighted Average Yield	6.00%
Weighted Average Life (Years)	4 years 1 month 6 days
Acquired Performing Loans | Ginnie Mae
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Unpaid Principal Balance	$ 30,000
Acquired Non-Performing Loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	617,062
Carrying Value	$ 490,222	565,387
Loan Count | loan	4,703
Weighted Average Yield	7.30%
Weighted Average Life (Years)	3 years 3 months 18 days
Acquired Non-Performing Loans | Ginnie Mae
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Unpaid Principal Balance	$ 26,100
Total Residential Mortgage Loans, held-for-sale
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	3,775,549
Carrying Value	3,519,739	6,042,664
Acquired Performing Loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	1,245,660
Carrying Value	$ 1,075,996	3,024,288
Loan Count | loan	9,258
Weighted Average Yield	4.80%
Weighted Average Life (Years)	8 years
Originated Loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	$ 1,675,955
Carrying Value	$ 1,748,913	1,589,324
Loan Count | loan	6,111
Weighted Average Yield	3.30%
Weighted Average Life (Years)	27 years 2 months 12 days
Total Residential Mortgage Loans, held-for-sale, at fair value
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	$ 2,921,615
Carrying Value	$ 2,824,909	$ 4,613,612
Loan Count | loan	15,369
Weighted Average Yield	3.90%
Weighted Average Life (Years)	19 years