Fair Value of Financial Instruments - Market Valuation Adjustments, Net (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2020	Dec. 31, 2019	Dec. 31, 2018
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
MSR (loss) income, net	$ (9,694)	$ 3,521	$ 7,076
Gain on re-measurement of 5 Arches investment	0	2,441	0
Total Market Valuation (Losses) Gains, Net	(541,399)	97,006	24,069
Mortgage Banking Activities, Net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	66,482	69,326	57,000
Mortgage Banking Activities, Net | Residential Loans
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(15,477)	3,267	23,144
Mortgage Banking Activities, Net | Residential loan purchase and forward sale commitments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	56,761	60,260	(1,336)
Mortgage Banking Activities, Net | Single-family rental loans held-for-sale, at fair value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	82,169	15,043	375
Mortgage Banking Activities, Net | Single-family rental loan purchase and interest rate lock commitments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	341	1,961	78
Mortgage Banking Activities, Net | Bridge loans
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(4,998)	4,518	0
Mortgage Banking Activities, Net | Risk management derivatives, net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(47,779)	(15,723)	34,739
Mortgage Banking Activities, Net | Trading securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(4,535)	0	0
Investment Fair Value Changes, Net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(588,438)	35,500	(25,689)
Investment Fair Value Changes, Net | Risk management derivatives, net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(59,142)	(127,169)	9,740
Investment Fair Value Changes, Net | Residential loans held-for-investment at Redwood
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(93,314)	58,891	(29,573)
Investment Fair Value Changes, Net | Single-family rental loans held-for-investment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(20,806)	272	0
Investment Fair Value Changes, Net | Bridge loans held-for-investment
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(10,629)	(2,139)	(29)
Investment Fair Value Changes, Net | Trading securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(226,196)	56,046	(8,055)
Investment Fair Value Changes, Net | Servicer advance investments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(8,901)	3,001	(701)
Investment Fair Value Changes, Net | Excess MSRs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(8,302)	(3,260)	1,823
Investment Fair Value Changes, Net | Net investments in Legacy Sequoia entities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(1,513)	(1,545)	(1,016)
Investment Fair Value Changes, Net | Net investments in Sequoia Choice entities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(13,244)	6,947	443
Investment Fair Value Changes, Net | Net investments in Freddie Mac SLST entities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(21,160)	27,206	1,271
Investment Fair Value Changes, Net | Net investments in Freddie Mac K-Series entities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(81,039)	21,430	931
Investment Fair Value Changes, Net | Net investments in CAFL entities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(36,754)	(3,636)	0
Investment Fair Value Changes, Net | Other investments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(7,050)	(544)	(434)
Investment Fair Value Changes, Net | Credit losses on AFS securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Changes in fair value of assets	(388)	0	(89)
Other Income
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
MSR (loss) income, net	(19,443)	(7,820)	(7,242)
Gain on re-measurement of 5 Arches investment	0	2,441	0
Other Income | Risk management derivatives, net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
MSR (loss) income, net	13,966	8,595	(4,734)
Other Income | MSRs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
MSR (loss) income, net	$ (33,409)	$ (18,856)	$ (2,508)