OTHER ASSETS AND LIABILITIES, GENERAL AND ADMINISTRATIVE, AND OTHER ITEMS - Schedule of Accretion and Other Amortization (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2021	Mar. 31, 2020
Accretion and other amortization:
Accretion of net discount on securities and loans	$ 12,007	$ 40,052
Accretion of servicer advances receivable discount and servicer advance investments	6,848	(10,915)
Accretion of excess mortgage servicing rights income	12,231	13,226
Amortization of deferred financing costs	(4,601)	(1,136)
Total accretion and other amortization	26,034	41,104
Secured Notes and Bonds Payable
Accretion and other amortization:
Amortization of discount on secured notes and bonds payable	(1)	(123)
Corporate Debt
Accretion and other amortization:
Amortization of discount on secured notes and bonds payable	$ (450)	$ 0