INVESTMENTS IN RESIDENTIAL MORTGAGE LOANS - Carrying Value of Mortgage Loans (Details) $ in Thousands	6 Months Ended
	Jun. 30, 2020 USD ($)
SEC Schedule, 12-29, Real Estate Companies, Investment in Movement in Mortgage Loans on Real Estate [Roll Forward]
Balance, beginning	$ 6,042,664
Transfer of loans to real estate owned	(28,197)
Balance, ending	3,519,739
Residential Portfolio Segment
SEC Schedule, 12-29, Real Estate Companies, Investment in Movement in Mortgage Loans on Real Estate [Roll Forward]
Balance, beginning	6,968,370
Fair value adjustment due to fair value option	(6,020)
Originations	19,252,135
Sales	(22,542,652)
Purchases/additional fundings	1,147,328
Proceeds from repayments	(242,096)
Transfer of loans to other assets	(15,309)
Transfer of loans to real estate owned	(23,889)
Transfers of loans to held for sale	0
Valuation provision on loans	(102,550)
Fair value adjustments due to, changes in instrument-specific credit risk	(39,589)
Fair value adjustments due to, other factors	(125,657)
Balance, ending	4,270,071
Residential Mortgage Loans, Held-for-Investment, at Fair Value | Residential Portfolio Segment
SEC Schedule, 12-29, Real Estate Companies, Investment in Movement in Mortgage Loans on Real Estate [Roll Forward]
Balance, beginning	925,706
Fair value adjustment due to fair value option	(6,020)
Originations	0
Sales	0
Purchases/additional fundings	0
Proceeds from repayments	(64,802)
Transfer of loans to other assets	0
Transfer of loans to real estate owned	(4,021)
Transfers of loans to held for sale	(59,681)
Valuation provision on loans	0
Fair value adjustments due to, changes in instrument-specific credit risk	6,923
Fair value adjustments due to, other factors	(47,773)
Balance, ending	750,332
Residential Mortgage Loans, Held-for-Sale | Acquired Loans, at Lower Cost or Fair Value | Residential Portfolio Segment
SEC Schedule, 12-29, Real Estate Companies, Investment in Movement in Mortgage Loans on Real Estate [Roll Forward]
Balance, beginning	1,429,052
Fair value adjustment due to fair value option	0
Originations	0
Sales	(642,644)
Purchases/additional fundings	110,741
Proceeds from repayments	(81,699)
Transfer of loans to other assets	(1,793)
Transfer of loans to real estate owned	(16,277)
Transfers of loans to held for sale	0
Valuation provision on loans	(102,550)
Fair value adjustments due to, changes in instrument-specific credit risk	0
Fair value adjustments due to, other factors	0
Balance, ending	694,830
Residential Mortgage Loans, Held-for-Sale | Acquired Loans | Residential Portfolio Segment
SEC Schedule, 12-29, Real Estate Companies, Investment in Movement in Mortgage Loans on Real Estate [Roll Forward]
Balance, beginning	4,613,612
Fair value adjustment due to fair value option	0
Originations	19,252,135
Sales	(21,900,008)
Purchases/additional fundings	1,036,587
Proceeds from repayments	(95,595)
Transfer of loans to other assets	(13,516)
Transfer of loans to real estate owned	(3,591)
Transfers of loans to held for sale	59,681
Valuation provision on loans	0
Fair value adjustments due to, changes in instrument-specific credit risk	(46,512)
Fair value adjustments due to, other factors	(77,884)
Balance, ending	$ 2,824,909