Fair Value of Financial Instruments - Portion of Net Gains (Losses) Attributable to Level 3 Assets and Liabilities Still Held Included in Net Income (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2016	Dec. 31, 2015	Dec. 31, 2014
Loan purchase commitments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 liability	$ (486)	$ 0	$ 0
Asset-backed securities issued, net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 liability	8,275	(8,366)	0
Commercial Secured Borrowings
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 liability	0	3,011	2,033
Residential Loans Held For Investment at Fair Value | Residential loans at Redwood
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	(17,370)	(5,541)	16,512
Residential Loans Held For Investment at Fair Value | Residential loans at consolidated Sequoia entities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	(14,391)	7,422	0
Commercial loans, at fair value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	0	(2,620)	3,357
Trading Securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	7,184	(13,391)	(25,216)
AFS Securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	(368)	(246)	(434)
MSRs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	42,964	(3,471)	(15,239)
Loan purchase commitments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	0	4,252	1,119
Guarantee Asset
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	$ (1,605)	$ (1,504)	$ 0