Residential Loans - Geographic Concentration of Residential Loans Recorded on Consolidated Balance Sheet (Details) - Residential loans, held-for-investment	12 Months Ended
	Dec. 31, 2017	Dec. 31, 2016
Concentration Risk [Line Items]
Concentration risk percentage	100.00%	100.00%
California
Concentration Risk [Line Items]
Concentration risk percentage	35.00%	40.00%
Texas
Concentration Risk [Line Items]
Concentration risk percentage	7.00%	9.00%
Washington
Concentration Risk [Line Items]
Concentration risk percentage	9.00%	8.00%
Florida
Concentration Risk [Line Items]
Concentration risk percentage	4.00%	3.00%
Georgia
Concentration Risk [Line Items]
Concentration risk percentage	2.00%	2.00%
New York
Concentration Risk [Line Items]
Concentration risk percentage	2.00%	2.00%
Other states (none greater than 5%)
Concentration Risk [Line Items]
Concentration risk percentage	41.00%	36.00%
Legacy Sequoia
Concentration Risk [Line Items]
Concentration risk percentage	100.00%	100.00%
Legacy Sequoia | California
Concentration Risk [Line Items]
Concentration risk percentage	18.00%	18.00%
Legacy Sequoia | Texas
Concentration Risk [Line Items]
Concentration risk percentage	5.00%	6.00%
Legacy Sequoia | Washington
Concentration Risk [Line Items]
Concentration risk percentage	2.00%	2.00%
Legacy Sequoia | Florida
Concentration Risk [Line Items]
Concentration risk percentage	13.00%	14.00%
Legacy Sequoia | Georgia
Concentration Risk [Line Items]
Concentration risk percentage	5.00%	5.00%
Legacy Sequoia | New York
Concentration Risk [Line Items]
Concentration risk percentage	9.00%	8.00%
Legacy Sequoia | Other states (none greater than 5%)
Concentration Risk [Line Items]
Concentration risk percentage	48.00%	47.00%
Sequoia Choice
Concentration Risk [Line Items]
Concentration risk percentage	100.00%	0.00%
Sequoia Choice | California
Concentration Risk [Line Items]
Concentration risk percentage	39.00%	0.00%
Sequoia Choice | Texas
Concentration Risk [Line Items]
Concentration risk percentage	10.00%	0.00%
Sequoia Choice | Washington
Concentration Risk [Line Items]
Concentration risk percentage	7.00%	0.00%
Sequoia Choice | Florida
Concentration Risk [Line Items]
Concentration risk percentage	4.00%	0.00%
Sequoia Choice | Georgia
Concentration Risk [Line Items]
Concentration risk percentage	3.00%	0.00%
Sequoia Choice | New York
Concentration Risk [Line Items]
Concentration risk percentage	3.00%	0.00%
Sequoia Choice | Other states (none greater than 5%)
Concentration Risk [Line Items]
Concentration risk percentage	34.00%	0.00%
Redwood
Concentration Risk [Line Items]
Concentration risk percentage	100.00%	100.00%
Redwood | California
Concentration Risk [Line Items]
Concentration risk percentage	45.00%	42.00%
Redwood | Texas
Concentration Risk [Line Items]
Concentration risk percentage	9.00%	10.00%
Redwood | Washington
Concentration Risk [Line Items]
Concentration risk percentage	5.00%	4.00%
Redwood | Florida
Concentration Risk [Line Items]
Concentration risk percentage	5.00%	5.00%
Redwood | Georgia
Concentration Risk [Line Items]
Concentration risk percentage	1.00%	1.00%
Redwood | New York
Concentration Risk [Line Items]
Concentration risk percentage	4.00%	4.00%
Redwood | Other states (none greater than 5%)
Concentration Risk [Line Items]
Concentration risk percentage	31.00%	34.00%