Residential Loans - Summary of Classifications and Carrying Value of Residential Loans (Details) - USD ($) $ in Thousands	Dec. 31, 2020	Dec. 31, 2019
MSRs
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Mortgage servicing rights	$ 172,000
Residential loans, held-for-investment | Legacy Sequoia
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Loans, at fair value	285,935	$ 407,890
Residential loans, held-for-investment | Sequoia Choice
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Loans, at fair value	1,565,322	2,291,463
Residential loans, held-for-investment | Freddie Mac SLST
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Loans, at fair value	2,221,153	2,367,215
Redwood | Residential loans, held-for-investment
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Loans, at fair value	0	2,111,897
Residential Loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Loans, at fair value	4,249,051	7,714,850
Residential Loans | Legacy Sequoia
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Loans, at fair value	285,935	407,890
Residential Loans | Sequoia Choice
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Loans, at fair value	1,565,322	2,291,463
Residential Loans | Freddie Mac SLST
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Loans, at fair value	2,221,153	2,367,215
Residential Loans | Held-for-sale at fair value
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Loans, at fair value	176,641	536,385
Residential Loans | Held-for-sale at fair value | Legacy Sequoia
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Loans, at fair value	0	0
Residential Loans | Held-for-sale at fair value | Sequoia Choice
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Loans, at fair value	0	0
Residential Loans | Held-for-sale at fair value | Freddie Mac SLST
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Loans, at fair value	0	0
Residential Loans | Residential loans, held-for-investment
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Loans, at fair value	4,072,410	7,178,465
Residential Loans | Residential loans, held-for-investment | Legacy Sequoia
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Loans, at fair value	285,935	407,890
Residential Loans | Residential loans, held-for-investment | Sequoia Choice
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Loans, at fair value	1,565,322	2,291,463
Residential Loans | Residential loans, held-for-investment | Freddie Mac SLST
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Loans, at fair value	2,221,153	2,367,215
Residential Loans | Redwood
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Loans, at fair value	176,641	2,648,282
Residential Loans | Redwood | Held-for-sale at fair value
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Loans, at fair value	176,641	536,385
Residential Loans | Redwood | Residential loans, held-for-investment
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Loans, at fair value	$ 0	$ 2,111,897