Schedule IV - Mortgage Loans On Real Estate (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2017 USD ($)	Dec. 31, 2017 USD ($) loan	Dec. 31, 2017 USD ($) Contract	Dec. 31, 2016 USD ($) loan
Residential loans, held-for-investment
Mortgage Loans on Real Estate [Line Items]
Carrying Amount	$ 3,687,265	$ 3,687,265	$ 3,687,265
Principal Amount Subject to Delinquent Principal or Interest	25,457	$ 25,457	25,457
Residential Loans Held For Sale
Mortgage Loans on Real Estate [Line Items]
Number of Loans | loan		2,013		1,121
Carrying Amount	1,427,945	$ 1,427,945	1,427,945
Principal Amount Subject to Delinquent Principal or Interest	459	$ 459	459	$ 300
Residential Loans Held For Sale | Hybrid ARM Loans
Mortgage Loans on Real Estate [Line Items]
Number of Loans | loan		93		173
Carrying Amount	79,347	$ 79,347	79,347
Principal Amount Subject to Delinquent Principal or Interest	$ 0	$ 0	0	$ 0
Residential Loans Held For Sale | Hybrid ARM Loans | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	2.88%			2.50%
Residential Loans Held For Sale | Hybrid ARM Loans | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	4.00%			6.00%
Residential Loans Held For Sale | Fixed Loans
Mortgage Loans on Real Estate [Line Items]
Number of Loans | loan		1,917		942
Carrying Amount	$ 1,348,300	$ 1,348,300	1,348,300
Principal Amount Subject to Delinquent Principal or Interest	$ 459	$ 459	459	$ 0
Residential Loans Held For Sale | Fixed Loans | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	2.88%			2.99%
Residential Loans Held For Sale | Fixed Loans | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	6.25%			6.25%
Residential Loans Held For Sale | ARM Loans
Mortgage Loans on Real Estate [Line Items]
Number of Loans | loan		3		6
Carrying Amount	$ 298	$ 298	298
Principal Amount Subject to Delinquent Principal or Interest	$ 0	$ 0	0	$ 300
Residential Loans Held For Sale | ARM Loans | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	1.50%			1.88%
Residential Loans Held For Sale | ARM Loans | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	3.00%			2.75%
Legacy Sequoia | Residential loans, held-for-investment
Mortgage Loans on Real Estate [Line Items]
Number of Loans | loan		3,178		3,735
Principal Amount Subject to Delinquent Principal or Interest	$ 25,457	$ 25,457	25,457	$ 30,166
Legacy Sequoia | Residential loans, held-for-investment | Hybrid ARM Loans
Mortgage Loans on Real Estate [Line Items]
Number of Loans | loan		33		37
Carrying Amount	13,039	$ 13,039	13,039
Principal Amount Subject to Delinquent Principal or Interest	$ 641	$ 641	641	$ 0
Legacy Sequoia | Residential loans, held-for-investment | Hybrid ARM Loans | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	2.63%
Legacy Sequoia | Residential loans, held-for-investment | Hybrid ARM Loans | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	3.75%
Legacy Sequoia | Residential loans, held-for-investment | ARM Loans
Mortgage Loans on Real Estate [Line Items]
Number of Loans | loan		3,145		3,698
Carrying Amount	$ 619,777	$ 619,777	619,777
Principal Amount Subject to Delinquent Principal or Interest	$ 24,816	$ 24,816	$ 24,816	$ 30,166
Legacy Sequoia | Residential loans, held-for-investment | ARM Loans | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	1.00%
Legacy Sequoia | Residential loans, held-for-investment | ARM Loans | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	5.63%
Sequoia Choice | Residential loans, held-for-investment | Fixed Loans
Mortgage Loans on Real Estate [Line Items]
Number of Loans		806	806
Carrying Amount	$ 620,062	$ 620,062	$ 620,062
Principal Amount Subject to Delinquent Principal or Interest	$ 0	$ 0	0
Sequoia Choice | Residential loans, held-for-investment | Fixed Loans | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	2.75%
Sequoia Choice | Residential loans, held-for-investment | Fixed Loans | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	6.50%
Redwood | Residential loans, held-for-investment
Mortgage Loans on Real Estate [Line Items]
Number of Loans | loan		3,292		3,068
Principal Amount Subject to Delinquent Principal or Interest	$ 0	$ 0	0	$ 237
Redwood | Residential loans, held-for-investment | Hybrid ARM Loans
Mortgage Loans on Real Estate [Line Items]
Number of Loans | loan		259		11
Carrying Amount	204,771	$ 204,771	204,771
Principal Amount Subject to Delinquent Principal or Interest	$ 0	$ 0	0	$ 0
Redwood | Residential loans, held-for-investment | Hybrid ARM Loans | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	2.63%
Redwood | Residential loans, held-for-investment | Hybrid ARM Loans | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	6.00%
Redwood | Residential loans, held-for-investment | Fixed Loans
Mortgage Loans on Real Estate [Line Items]
Number of Loans | loan		3,033		3,057
Carrying Amount	$ 2,229,616	$ 2,229,616	2,229,616
Principal Amount Subject to Delinquent Principal or Interest	$ 0	$ 0	$ 0	$ 237
Redwood | Residential loans, held-for-investment | Fixed Loans | Minimum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	2.75%
Redwood | Residential loans, held-for-investment | Fixed Loans | Maximum
Mortgage Loans on Real Estate [Line Items]
Interest Rate	6.75%