Fair Value of Financial Instruments - Portion of Net Gains (Losses) Attributable to Level 3 Assets and Liabilities Still Held Included in Net Income (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2020	Dec. 31, 2019	Dec. 31, 2018
Loan purchase and interest rate lock commitments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 liability	$ (577)	$ (1,290)	$ (732)
Contingent Consideration
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 liability	0	(3,217)	0
Residential loans, held-for-investment | Residential loans at Redwood
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	1,138	67,470	(17,757)
Residential loans, held-for-investment | Net investments in consolidated Sequoia entities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	(14,646)	4,529	(1,046)
Residential loans, held-for-investment | Freddie Mac SLST
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	(21,220)	27,225	21,295
Business purpose loans
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	9,420	14,603	445
Multifamily securities | Freddie Mac K-Series
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	(9,309)	21,430	931
Single-family rental loans | CAFL
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	(37,062)	(14,681)	0
Trading securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	(83,327)	18,865	(12,256)
AFS Securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	(388)	0	(89)
Servicer advance investments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	(8,902)	3,001	(702)
MSRs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	(17,545)	(11,957)	1,942
Excess MSRs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	(8,302)	(3,260)	1,824
Shared home appreciation options
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	(1,884)	842	0
Loan purchase and interest rate lock commitments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	$ 15,027	$ 10,190	$ 2,913