INVESTMENTS IN RESIDENTIAL MORTGAGE LOANS - Residential Mortgage Loans Outstanding by Loan Type, Excluding REO (Details) $ in Thousands	9 Months Ended
	Sep. 30, 2020 USD ($) loan	Dec. 31, 2019 USD ($)
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	$ 1,261,905
Weighted Average Life (Years)	2 years 3 months 18 days
Residential Mortgage Loans, Held-for-Investment, at Fair Value
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	$ 784,838
Carrying Value	$ 718,802	$ 925,706
Loan Count | loan	12,587
Weighted Average Yield	6.70%
Weighted Average Life (Years)	4 years 2 months 12 days
Acquired Residential Mortgage Loans Held-for-Sale
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	$ 719,533
Carrying Value	$ 578,353	1,429,052
Loan Count | loan	7,529
Weighted Average Yield	7.20%
Weighted Average Life (Years)	3 years 3 months 18 days
Acquired Reverse Mortgage Loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	$ 11,784
Carrying Value	$ 5,848	5,844
Loan Count | loan	28
Weighted Average Yield	7.80%
Weighted Average Life (Years)	3 years 10 months 24 days
Interest in reverse mortgage loans	70.00%
Unpaid principal balance	$ 600
Percentage of loans that have reached a termination event	47.00%
Acquired Reverse Mortgage Loans | Mr. Cooper
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest in reverse mortgage loans	30.00%
Acquired Performing Loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	$ 205,746
Carrying Value	$ 191,580	857,821
Loan Count | loan	4,138
Weighted Average Yield	6.60%
Weighted Average Life (Years)	3 years 4 months 24 days
Acquired Performing Loans | Ginnie Mae
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Unpaid Principal Balance	$ 490,900
Acquired Non-Performing Loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	502,003
Carrying Value	$ 380,925	565,387
Loan Count | loan	3,363
Weighted Average Yield	7.50%
Weighted Average Life (Years)	3 years 3 months 18 days
Acquired Non-Performing Loans | Ginnie Mae
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Unpaid Principal Balance	$ 21,200
Acquired Performing Loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	1,205,668
Carrying Value	$ 1,176,302	3,024,288
Loan Count | loan	7,590
Weighted Average Yield	4.20%
Weighted Average Life (Years)	9 years 10 months 24 days
Acquired non-performing loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	$ 436,936
Carrying Value	$ 338,451	0
Loan Count | loan	2,872
Weighted Average Yield	7.50%
Originated loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	$ 2,705,121
Carrying Value	$ 2,843,720	1,589,324
Loan Count | loan	9,947
Weighted Average Yield	2.90%
Weighted Average Life (Years)	22 years 1 month 6 days
Total Residential Mortgage Loans, held-for-sale, at fair value
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	$ 4,347,725
Carrying Value	$ 4,358,473	4,613,612
Loan Count | loan	20,409
Weighted Average Yield	3.70%
Weighted Average Life (Years)	16 years 9 months 18 days
Total Residential Mortgage Loans, held-for-sale
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	$ 5,067,258
Carrying Value	$ 4,936,826	$ 6,042,664