Fair Value of Financial Instruments - Portion of Net Gains (Losses) Attributable to Level 3 Assets and Liabilities Still Held Included in Net Income (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2017	Dec. 31, 2016	Dec. 31, 2015
Loan purchase commitments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 liability	$ (3,706)	$ (486)	$ 0
Commercial Secured Borrowings
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 liability	0	0	3,011
ABS issued
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 liability	(29,187)	8,275	(8,366)
Residential Loans Held For Investment at Fair Value | Residential loans at Redwood
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	523	(17,370)	(5,541)
Residential Loans Held For Investment at Fair Value | Residential loans at consolidated Sequoia entities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	17,727	(14,391)	7,422
Commercial loans, at fair value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	0	0	(2,620)
Trading Securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	28,612	7,184	(13,391)
AFS Securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	(1,011)	(368)	(246)
MSRs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	1,277	42,964	(3,471)
Loan purchase commitments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	3,243	0	4,252
Loan forward sale commitments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	2,177	0	0
Other assets - Guarantee asset
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	$ (1,223)	$ (1,605)	$ (1,504)