Asset-Backed Securities Issued - Summary of Carrying Value Components of Collateral for Asset-Backed Securities Issued and Outstanding (Details) - USD ($) $ in Thousands	Sep. 30, 2019	Dec. 31, 2018
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	$ 9,311,059	$ 5,992,049
Residential loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	5,488,698	3,822,009
Multifamily loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	3,791,622	2,144,598
Restricted cash
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	158	1,168
Accrued interest receivable
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	30,037	20,331
REO
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	544	3,943
Legacy Sequoia
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	430,478	524,869
Legacy Sequoia | Residential loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	429,159	519,958
Legacy Sequoia | Multifamily loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0	0
Legacy Sequoia | Restricted cash
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	143	146
Legacy Sequoia | Accrued interest receivable
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	716	822
Legacy Sequoia | REO
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	460	3,943
Sequoia Choice
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	2,629,137	2,089,392
Sequoia Choice | Residential loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	2,618,316	2,079,382
Sequoia Choice | Multifamily loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0	0
Sequoia Choice | Restricted cash
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	15	1,022
Sequoia Choice | Accrued interest receivable
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	10,806	8,988
Sequoia Choice | REO
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0	0
Freddie Mac SLST
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	2,448,522	1,226,595
Freddie Mac SLST | Residential loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	2,441,223	1,222,669
Freddie Mac SLST | Multifamily loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0	0
Freddie Mac SLST | Restricted cash
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0	0
Freddie Mac SLST | Accrued interest receivable
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	7,215	3,926
Freddie Mac SLST | REO
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	84	0
Freddie Mac K-Series
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	3,802,922	2,151,193
Freddie Mac K-Series | Residential loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0	0
Freddie Mac K-Series | Multifamily loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	3,791,622	2,144,598
Freddie Mac K-Series | Restricted cash
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0	0
Freddie Mac K-Series | Accrued interest receivable
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	11,300	6,595
Freddie Mac K-Series | REO
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	$ 0	$ 0