Asset-Backed Securities Issued - Summary of Carrying Value Components of Collateral for Asset-Backed Securities Issued and Outstanding (Details) - USD ($) $ in Thousands	Dec. 31, 2019	Dec. 31, 2018
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	$ 11,711,417	$ 5,992,049
Residential loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	5,066,568	3,822,009
Business purpose residential loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	2,192,552
Multifamily loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	4,408,524	2,144,598
Restricted cash
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	170	1,168
Accrued interest receivable
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	40,903	20,331
REO
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	2,700	3,943
Legacy Sequoia
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	409,148	524,869
Legacy Sequoia | Residential loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	407,890	519,958
Legacy Sequoia | Business purpose residential loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0
Legacy Sequoia | Multifamily loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0	0
Legacy Sequoia | Restricted cash
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	143	146
Legacy Sequoia | Accrued interest receivable
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	655	822
Legacy Sequoia | REO
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	460	3,943
Sequoia Choice
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	2,301,314	2,089,392
Sequoia Choice | Residential loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	2,291,463	2,079,382
Sequoia Choice | Business purpose residential loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0
Sequoia Choice | Multifamily loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0	0
Sequoia Choice | Restricted cash
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	27	1,022
Sequoia Choice | Accrued interest receivable
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	9,824	8,988
Sequoia Choice | REO
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0	0
Freddie Mac SLST
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	2,374,973	1,226,595
Freddie Mac SLST | Residential loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	2,367,215	1,222,669
Freddie Mac SLST | Business purpose residential loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0
Freddie Mac SLST | Multifamily loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0	0
Freddie Mac SLST | Restricted cash
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0	0
Freddie Mac SLST | Accrued interest receivable
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	7,313	3,926
Freddie Mac SLST | REO
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	445	0
Freddie Mac K-Series
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	4,422,063	2,151,193
Freddie Mac K-Series | Residential loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0	0
Freddie Mac K-Series | Business purpose residential loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0
Freddie Mac K-Series | Multifamily loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	4,408,524	2,144,598
Freddie Mac K-Series | Restricted cash
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0	0
Freddie Mac K-Series | Accrued interest receivable
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	13,539	6,595
Freddie Mac K-Series | REO
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0	0
CAFL
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	2,203,919	0
CAFL | Residential loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0	0
CAFL | Business purpose residential loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	2,192,552
CAFL | Multifamily loans
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0	0
CAFL | Restricted cash
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	0	0
CAFL | Accrued interest receivable
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	9,572	0
CAFL | REO
Financial Instruments Owned and Pledged as Collateral [Line Items]
Collateral for ABS Issued	$ 1,795	$ 0