Residential Loans - Geographic Concentration of Residential Loans Recorded on Consolidated Balance Sheet (Details) - Residential loans, held-for-investment	12 Months Ended
	Dec. 31, 2020	Dec. 31, 2019
Concentration Risk [Line Items]
Concentration risk percentage	100.00%	100.00%
California
Concentration Risk [Line Items]
Concentration risk percentage	43.00%	36.00%
Texas
Concentration Risk [Line Items]
Concentration risk percentage	10.00%	6.00%
Washington
Concentration Risk [Line Items]
Concentration risk percentage	7.00%	7.00%
Colorado
Concentration Risk [Line Items]
Concentration risk percentage	5.00%	6.00%
Florida
Concentration Risk [Line Items]
Concentration risk percentage	3.00%	4.00%
Illinois
Concentration Risk [Line Items]
Concentration risk percentage	3.00%
Maryland
Concentration Risk [Line Items]
Concentration risk percentage	2.00%
New Jersey
Concentration Risk [Line Items]
Concentration risk percentage	1.00%	2.00%
New York
Concentration Risk [Line Items]
Concentration risk percentage	1.00%	1.00%
Ohio
Concentration Risk [Line Items]
Concentration risk percentage	0.00%
Other states (none greater than 5%)
Concentration Risk [Line Items]
Concentration risk percentage	25.00%	38.00%
Legacy Sequoia
Concentration Risk [Line Items]
Concentration risk percentage	100.00%	100.00%
Legacy Sequoia | California
Concentration Risk [Line Items]
Concentration risk percentage	17.00%	18.00%
Legacy Sequoia | Texas
Concentration Risk [Line Items]
Concentration risk percentage	5.00%	6.00%
Legacy Sequoia | Washington
Concentration Risk [Line Items]
Concentration risk percentage	1.00%	1.00%
Legacy Sequoia | Colorado
Concentration Risk [Line Items]
Concentration risk percentage	3.00%	3.00%
Legacy Sequoia | Florida
Concentration Risk [Line Items]
Concentration risk percentage	14.00%	14.00%
Legacy Sequoia | Illinois
Concentration Risk [Line Items]
Concentration risk percentage	3.00%
Legacy Sequoia | Maryland
Concentration Risk [Line Items]
Concentration risk percentage	2.00%
Legacy Sequoia | New Jersey
Concentration Risk [Line Items]
Concentration risk percentage	5.00%	4.00%
Legacy Sequoia | New York
Concentration Risk [Line Items]
Concentration risk percentage	10.00%	10.00%
Legacy Sequoia | Ohio
Concentration Risk [Line Items]
Concentration risk percentage	5.00%
Legacy Sequoia | Other states (none greater than 5%)
Concentration Risk [Line Items]
Concentration risk percentage	35.00%	44.00%
Sequoia Choice
Concentration Risk [Line Items]
Concentration risk percentage	100.00%	100.00%
Sequoia Choice | California
Concentration Risk [Line Items]
Concentration risk percentage	34.00%	35.00%
Sequoia Choice | Texas
Concentration Risk [Line Items]
Concentration risk percentage	10.00%	9.00%
Sequoia Choice | Washington
Concentration Risk [Line Items]
Concentration risk percentage	5.00%	7.00%
Sequoia Choice | Colorado
Concentration Risk [Line Items]
Concentration risk percentage	3.00%	4.00%
Sequoia Choice | Florida
Concentration Risk [Line Items]
Concentration risk percentage	6.00%	5.00%
Sequoia Choice | Illinois
Concentration Risk [Line Items]
Concentration risk percentage	2.00%
Sequoia Choice | Maryland
Concentration Risk [Line Items]
Concentration risk percentage	1.00%
Sequoia Choice | New Jersey
Concentration Risk [Line Items]
Concentration risk percentage	2.00%	2.00%
Sequoia Choice | New York
Concentration Risk [Line Items]
Concentration risk percentage	5.00%	4.00%
Sequoia Choice | Ohio
Concentration Risk [Line Items]
Concentration risk percentage	0.00%
Sequoia Choice | Other states (none greater than 5%)
Concentration Risk [Line Items]
Concentration risk percentage	32.00%	34.00%
Freddie Mac SLST
Concentration Risk [Line Items]
Concentration risk percentage	100.00%	100.00%
Freddie Mac SLST | California
Concentration Risk [Line Items]
Concentration risk percentage	14.00%	14.00%
Freddie Mac SLST | Texas
Concentration Risk [Line Items]
Concentration risk percentage	3.00%	3.00%
Freddie Mac SLST | Washington
Concentration Risk [Line Items]
Concentration risk percentage	2.00%	2.00%
Freddie Mac SLST | Colorado
Concentration Risk [Line Items]
Concentration risk percentage	1.00%	0.00%
Freddie Mac SLST | Florida
Concentration Risk [Line Items]
Concentration risk percentage	10.00%	10.00%
Freddie Mac SLST | Illinois
Concentration Risk [Line Items]
Concentration risk percentage	5.00%
Freddie Mac SLST | Maryland
Concentration Risk [Line Items]
Concentration risk percentage	5.00%
Freddie Mac SLST | New Jersey
Concentration Risk [Line Items]
Concentration risk percentage	7.00%	7.00%
Freddie Mac SLST | New York
Concentration Risk [Line Items]
Concentration risk percentage	10.00%	10.00%
Freddie Mac SLST | Ohio
Concentration Risk [Line Items]
Concentration risk percentage	2.00%
Freddie Mac SLST | Other states (none greater than 5%)
Concentration Risk [Line Items]
Concentration risk percentage	41.00%	54.00%
Redwood
Concentration Risk [Line Items]
Concentration risk percentage	0.00%	100.00%
Redwood | California
Concentration Risk [Line Items]
Concentration risk percentage	0.00%	45.00%
Redwood | Texas
Concentration Risk [Line Items]
Concentration risk percentage	0.00%	8.00%
Redwood | Washington
Concentration Risk [Line Items]
Concentration risk percentage	0.00%	5.00%
Redwood | Colorado
Concentration Risk [Line Items]
Concentration risk percentage	0.00%	4.00%
Redwood | Florida
Concentration Risk [Line Items]
Concentration risk percentage	0.00%	5.00%
Redwood | Illinois
Concentration Risk [Line Items]
Concentration risk percentage	0.00%
Redwood | Maryland
Concentration Risk [Line Items]
Concentration risk percentage	0.00%
Redwood | New Jersey
Concentration Risk [Line Items]
Concentration risk percentage	0.00%	2.00%
Redwood | New York
Concentration Risk [Line Items]
Concentration risk percentage	0.00%	4.00%
Redwood | Ohio
Concentration Risk [Line Items]
Concentration risk percentage	0.00%
Redwood | Other states (none greater than 5%)
Concentration Risk [Line Items]
Concentration risk percentage	0.00%	27.00%