Fair Value of Financial Instruments - Portion of Net Gains (Losses) Attributable to Level 3 Assets and Liabilities Still Held Included in Net Income (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2020	Mar. 31, 2019
Loan purchase and interest rate lock commitments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 liabilities still held included in net income	$ (3,967)	$ (753)
Contingent Consideration
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 liabilities still held included in net income	(312)	0
ABS Issued
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 liabilities still held included in net income	360,640	(60,182)
Held-for-investment at fair value | Residential loans at Redwood
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	(102,867)	35,801
Held-for-investment at fair value | Residential loans at consolidated Sequoia entities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	(179,499)	14,472
Held-for-investment at fair value | Freddie Mac K-Series
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	(193,035)	23,527
Business purpose residential loans
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	(68,864)	1,050
Single family rental loans | CAFL
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	(271,917)	0
Multifamily loans | Freddie Mac K-Series
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	(10,351)	34,372
Trading securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	(136,359)	21,543
Servicer advance investments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	(6,062)	1,008
MSRs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	(16,640)	(4,295)
Excess MSRs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	(9,494)	(437)
Loan purchase and interest rate lock commitments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	0	4,962
Other assets - Guarantee asset
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	(781)	(277)
Shared home appreciation options
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	$ (7,554)	$ 0