OTHER ASSETS AND LIABILITIES, GENERAL AND ADMINISTRATIVE, AND OTHER ITEMS - Schedule of Accretion and Other Amortization (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2021	Jun. 30, 2020	Jun. 30, 2021	Jun. 30, 2020
Accretion and other amortization:
Accretion of net discount on securities and loans	$ 10,090	$ 22,739	$ 22,097	$ 62,791
Accretion of servicer advances receivable discount and servicer advance investments	(52)	24,379	6,796	13,464
Accretion of excess mortgage servicing rights income	(238)	(644)	11,993	12,582
Amortization of deferred financing costs	(3,776)	(4,856)	(8,377)	(5,992)
Total accretion and other amortization	5,583	39,886	31,617	80,990
Secured Notes and Bonds Payable
Accretion and other amortization:
Amortization of discount on secured notes and bonds payable	(3)	(123)	(3)	(246)
Corporate Debt
Accretion and other amortization:
Amortization of discount on secured notes and bonds payable	$ (438)	$ (1,609)	$ (889)	$ (1,609)