Residential Loans - Geographic Concentration of Residential Loans Recorded on Consolidated Balance Sheet (Details) - Residential loans, held-for-investment	12 Months Ended
	Dec. 31, 2019	Dec. 31, 2018
Concentration Risk [Line Items]
Concentration risk percentage	100.00%	100.00%
California
Concentration Risk [Line Items]
Concentration risk percentage	36.00%	40.00%
Washington
Concentration Risk [Line Items]
Concentration risk percentage	7.00%	10.00%
Texas
Concentration Risk [Line Items]
Concentration risk percentage	6.00%	6.00%
Colorado
Concentration Risk [Line Items]
Concentration risk percentage	6.00%
Florida
Concentration Risk [Line Items]
Concentration risk percentage	4.00%	4.00%
New Jersey
Concentration Risk [Line Items]
Concentration risk percentage	2.00%	2.00%
New York
Concentration Risk [Line Items]
Concentration risk percentage	1.00%	3.00%
Other states (none greater than 5%)
Concentration Risk [Line Items]
Concentration risk percentage	38.00%	35.00%
Legacy Sequoia
Concentration Risk [Line Items]
Concentration risk percentage	100.00%	100.00%
Legacy Sequoia | California
Concentration Risk [Line Items]
Concentration risk percentage	18.00%	19.00%
Legacy Sequoia | Washington
Concentration Risk [Line Items]
Concentration risk percentage	1.00%	1.00%
Legacy Sequoia | Texas
Concentration Risk [Line Items]
Concentration risk percentage	6.00%	6.00%
Legacy Sequoia | Colorado
Concentration Risk [Line Items]
Concentration risk percentage	3.00%
Legacy Sequoia | Florida
Concentration Risk [Line Items]
Concentration risk percentage	14.00%	13.00%
Legacy Sequoia | New Jersey
Concentration Risk [Line Items]
Concentration risk percentage	4.00%	4.00%
Legacy Sequoia | New York
Concentration Risk [Line Items]
Concentration risk percentage	10.00%	10.00%
Legacy Sequoia | Other states (none greater than 5%)
Concentration Risk [Line Items]
Concentration risk percentage	44.00%	47.00%
Sequoia Choice
Concentration Risk [Line Items]
Concentration risk percentage	100.00%	100.00%
Sequoia Choice | California
Concentration Risk [Line Items]
Concentration risk percentage	35.00%	39.00%
Sequoia Choice | Washington
Concentration Risk [Line Items]
Concentration risk percentage	7.00%	7.00%
Sequoia Choice | Texas
Concentration Risk [Line Items]
Concentration risk percentage	9.00%	8.00%
Sequoia Choice | Colorado
Concentration Risk [Line Items]
Concentration risk percentage	4.00%
Sequoia Choice | Florida
Concentration Risk [Line Items]
Concentration risk percentage	5.00%	4.00%
Sequoia Choice | New Jersey
Concentration Risk [Line Items]
Concentration risk percentage	2.00%	1.00%
Sequoia Choice | New York
Concentration Risk [Line Items]
Concentration risk percentage	4.00%	5.00%
Sequoia Choice | Other states (none greater than 5%)
Concentration Risk [Line Items]
Concentration risk percentage	34.00%	36.00%
Freddie Mac SLST
Concentration Risk [Line Items]
Concentration risk percentage	100.00%	100.00%
Freddie Mac SLST | California
Concentration Risk [Line Items]
Concentration risk percentage	14.00%	12.00%
Freddie Mac SLST | Washington
Concentration Risk [Line Items]
Concentration risk percentage	2.00%	2.00%
Freddie Mac SLST | Texas
Concentration Risk [Line Items]
Concentration risk percentage	3.00%	3.00%
Freddie Mac SLST | Colorado
Concentration Risk [Line Items]
Concentration risk percentage	0.00%
Freddie Mac SLST | Florida
Concentration Risk [Line Items]
Concentration risk percentage	10.00%	10.00%
Freddie Mac SLST | New Jersey
Concentration Risk [Line Items]
Concentration risk percentage	7.00%	7.00%
Freddie Mac SLST | New York
Concentration Risk [Line Items]
Concentration risk percentage	10.00%	10.00%
Freddie Mac SLST | Other states (none greater than 5%)
Concentration Risk [Line Items]
Concentration risk percentage	54.00%	56.00%
Redwood
Concentration Risk [Line Items]
Concentration risk percentage	100.00%	100.00%
Redwood | California
Concentration Risk [Line Items]
Concentration risk percentage	45.00%	47.00%
Redwood | Washington
Concentration Risk [Line Items]
Concentration risk percentage	5.00%	5.00%
Redwood | Texas
Concentration Risk [Line Items]
Concentration risk percentage	8.00%	8.00%
Redwood | Colorado
Concentration Risk [Line Items]
Concentration risk percentage	4.00%
Redwood | Florida
Concentration Risk [Line Items]
Concentration risk percentage	5.00%	5.00%
Redwood | New Jersey
Concentration Risk [Line Items]
Concentration risk percentage	2.00%	1.00%
Redwood | New York
Concentration Risk [Line Items]
Concentration risk percentage	4.00%	3.00%
Redwood | Other states (none greater than 5%)
Concentration Risk [Line Items]
Concentration risk percentage	27.00%	31.00%