INVESTMENTS IN RESIDENTIAL MORTGAGE LOANS - Loans Outstanding by Loan Type, Excluding REO (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2018 USD ($) loan	Dec. 31, 2017 USD ($) loan
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Weighted Average Life (Years)	10 months 29 days
UPB of Underlying Mortgages	$ 103,000,000
Performing Loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	636,874	$ 557,381
Carrying Value	$ 591,264	$ 507,615
Loan Count | loan	8,424	8,876
Weighted Average Yield	8.00%	8.00%
Weighted Average Life (Years)	4 years 9 months 11 days	5 years 6 months 1 day
Floating Rate Loans as a % of Face Amount	20.30%	22.10%
LTV Ratio	77.70%	76.40%
Weighted Average Delinquency	8.90%	8.70%
Weighted Average FICO	649	649
Performing Loans | Ginnie Mae EBO
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
UPB of Underlying Mortgages	$ 24,300	$ 33,700
Purchased Credit Deteriorated Loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	191,497	249,254
Carrying Value	$ 144,065	$ 183,540
Loan Count | loan	1,556	2,142
Weighted Average Yield	7.60%	7.20%
Weighted Average Life (Years)	3 years 1 month 19 days	3 years 1 month 13 days
Floating Rate Loans as a % of Face Amount	16.40%	14.70%
LTV Ratio	84.60%	84.20%
Weighted Average Delinquency	71.50%	75.80%
Weighted Average FICO	596	597
Total Residential Mortgage Loans, Held-for-Investment
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	$ 828,371	$ 806,635
Carrying Value	$ 735,329	$ 691,155
Loan Count | loan	9,980	11,018
Weighted Average Yield	7.90%	7.70%
Weighted Average Life (Years)	4 years 4 months 25 days	4 years 9 months 6 days
Floating Rate Loans as a % of Face Amount	19.40%	19.80%
LTV Ratio	79.30%	78.80%
Weighted Average Delinquency	23.30%	29.40%
Weighted Average FICO	637	633
Reverse Mortgage Loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	$ 13,807	$ 16,755
Carrying Value	$ 6,557	$ 6,870
Loan Count | loan	37	48
Weighted Average Yield	8.10%	7.50%
Weighted Average Life (Years)	4 years 9 months 27 days	4 years 6 months 12 days
Floating Rate Loans as a % of Face Amount	10.60%	15.90%
LTV Ratio	142.50%	141.20%
Weighted Average Delinquency	67.80%	77.80%
Performing Loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	$ 408,724	$ 1,044,116
Carrying Value	$ 413,883	$ 1,071,371
Loan Count | loan	7,144	15,464
Weighted Average Yield	4.40%	4.00%
Weighted Average Life (Years)	3 years 10 months 20 days	4 years 10 months 2 days
Floating Rate Loans as a % of Face Amount	56.60%	10.20%
LTV Ratio	61.30%	53.20%
Weighted Average Delinquency	9.00%	7.00%
Weighted Average FICO	670	654
Non-Performing Loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	$ 621,700	$ 846,181
Carrying Value	$ 512,040	$ 647,293
Loan Count | loan	5,029	5,597
Weighted Average Yield	5.50%	5.60%
Weighted Average Life (Years)	3 years 15 days	4 years 3 months 12 days
Floating Rate Loans as a % of Face Amount	14.90%	18.70%
LTV Ratio	88.10%	94.40%
Weighted Average Delinquency	72.60%	63.30%
Weighted Average FICO	588	581
Total Residential Mortgage Loans, Held-for-Sale
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	$ 1,044,231	$ 1,907,052
Carrying Value	$ 932,480	$ 1,725,534
Loan Count | loan	12,210	21,109
Weighted Average Yield	5.10%	4.80%
Weighted Average Life (Years)	3 years 4 months 23 days	4 years 7 months 2 days
Floating Rate Loans as a % of Face Amount	31.20%	14.00%
LTV Ratio	78.30%	72.20%
Weighted Average Delinquency	47.60%	32.60%
Weighted Average FICO	621	622
Acquired Loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	$ 2,295,340
Carrying Value	$ 2,153,269
Loan Count | loan	12,873
Weighted Average Yield	4.50%
Weighted Average Life (Years)	8 years 16 days
Floating Rate Loans as a % of Face Amount	7.70%
LTV Ratio	75.70%
Weighted Average Delinquency	14.00%
Weighted Average FICO	626
Originated Loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	$ 638,173
Carrying Value	$ 655,260
Loan Count | loan	2,307
Weighted Average Yield	5.20%
Weighted Average Life (Years)	28 years 6 months 15 days
Floating Rate Loans as a % of Face Amount	96.30%
LTV Ratio	80.00%
Weighted Average Delinquency	3.80%
Weighted Average FICO	714
Total Residential Mortgage Loans, held-for-sale, at fair value
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Outstanding Face Amount	$ 2,933,513
Carrying Value	$ 2,808,529
Loan Count | loan	15,180
Weighted Average Yield	4.60%
Weighted Average Life (Years)	12 years 6 months 1 day
Floating Rate Loans as a % of Face Amount	27.00%
LTV Ratio	76.60%
Weighted Average Delinquency	11.80%
Weighted Average FICO	645
Reverse Mortgage Loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest in reverse mortgage loans	70.00%
Unpaid principal balance	$ 500	$ 500
Percentage of loans that have reached a termination event	54.90%	54.30%
Non-Performing Loans | Ginnie Mae EBO
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
UPB of Underlying Mortgages	$ 51,900	$ 66,500
Non Agency Mortgage Loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
UPB of Underlying Mortgages	$ 122,300
Nationstar | Reverse Mortgage Loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest in reverse mortgage loans	30.00%