Asset-Backed Securities Issued - Summary of Carrying Value Components of Collateral for Asset-Backed Securities Issued and Outstanding (Details) - USD ($) $ in Thousands	Dec. 31, 2016	Dec. 31, 2015
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	$ 798,317	$ 1,195,574
Residential loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	791,636	1,021,870
Commercial loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0	166,016
Restricted cash
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	148	365
Accrued interest receivable
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	1,000	2,428
REO
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	5,533	4,895
Sequoia Entities
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	798,317	1,028,124
Sequoia Entities | Residential loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	791,636	1,021,870
Sequoia Entities | Commercial loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0	0
Sequoia Entities | Restricted cash
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	148	228
Sequoia Entities | Accrued interest receivable
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	1,000	1,131
Sequoia Entities | REO
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	5,533	4,895
Commercial Securitization
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0	167,450
Commercial Securitization | Residential loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0	0
Commercial Securitization | Commercial loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0	166,016
Commercial Securitization | Restricted cash
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0	137
Commercial Securitization | Accrued interest receivable
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0	1,297
Commercial Securitization | REO
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	$ 0	$ 0