Borrowings - Schedule of Financing Liabilities (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2019	Dec. 31, 2018
Debt Instrument [Line Items]
HMBS-related borrowings	$ 6,063,435	$ 5,380,448
Other financing liabilities	972,595	1,062,090
Long-term debt, gross	1,030,306	452,736
Financing Liabilities
Debt Instrument [Line Items]
HMBS-related borrowings	6,063,435	5,380,448
Other financing liabilities	950,593	1,032,856
Long-term debt, gross	7,036,030	6,442,538
IndyMac Mortgage Loan Trust (INDX 2004-AR11) | Financing Liabilities
Debt Instrument [Line Items]
Other financing liabilities	9,794	11,012
Residential Asset Securitization Trust 2003-A11 (RAST 2003-A11) | Financing Liabilities
Debt Instrument [Line Items]
Other financing liabilities	12,208	13,803
Financing Liability Owed to Securitization Investors | Financing Liabilities
Debt Instrument [Line Items]
Other financing liabilities	$ 22,002	24,815
Financing Liability – Advances Pledged
Debt Instrument [Line Items]
Interest rate (percentage)	4.50%
Financing Liability – Advances Pledged | Financing Liabilities
Debt Instrument [Line Items]
Other financing liabilities	$ 0	4,419
LIBOR | Financing Liabilities
Debt Instrument [Line Items]
Interest rate (percentage)	2.60%
LIBOR | Residential Asset Securitization Trust 2003-A11 (RAST 2003-A11)
Debt Instrument [Line Items]
Interest rate (percentage)	0.45%
Original Rights to MSRs Agreements | Financing Liabilities
Debt Instrument [Line Items]
Other financing liabilities	$ 603,046	436,511
2017 Agreements and New RMSR Agreements | Financing Liabilities
Debt Instrument [Line Items]
Other financing liabilities	35,445	138,854
PHH MSR Agreements | Financing Liabilities
Debt Instrument [Line Items]
Other financing liabilities	$ 312,102	$ 457,491