Principles of Consolidation - Assets and Liabilities of Consolidated Variable Interest Entity's (Details) $ in Thousands	Dec. 31, 2019 USD ($) investment	Dec. 31, 2018 USD ($) investment
Variable Interest Entity [Line Items]
Assets	$ 11,931,869	$ 6,331,191
Liabilities	$ 10,717,072	$ 5,709,807
Number of VIEs | investment	49	33
Residential loans, held-for-investment
Variable Interest Entity [Line Items]
Assets	$ 5,066,568	$ 3,822,009
Business purpose residential loans, held-for-investment
Variable Interest Entity [Line Items]
Assets	2,192,552
Multifamily loans, held-for-investment
Variable Interest Entity [Line Items]
Assets	4,408,524	2,144,598
Other investments
Variable Interest Entity [Line Items]
Assets	184,802	312,688
Cash and cash equivalents
Variable Interest Entity [Line Items]
Assets	9,015
Restricted cash
Variable Interest Entity [Line Items]
Assets	21,936	26,531
Accrued interest receivable
Variable Interest Entity [Line Items]
Assets	45,772	21,422
Other assets
Variable Interest Entity [Line Items]
Assets	2,700	3,943
Short-term debt
Variable Interest Entity [Line Items]
Liabilities	152,554	262,740
Accrued interest payable
Variable Interest Entity [Line Items]
Liabilities	34,060	17,380
Accrued expenses and other liabilities
Variable Interest Entity [Line Items]
Liabilities	14,983	19,614
Asset-backed securities issued
Variable Interest Entity [Line Items]
Liabilities	10,515,475	5,410,073
Legacy Sequoia
Variable Interest Entity [Line Items]
Assets	409,148	524,869
Liabilities	$ 402,860	$ 512,811
Number of VIEs | investment	20	20
Legacy Sequoia | Residential loans, held-for-investment
Variable Interest Entity [Line Items]
Assets	$ 407,890	$ 519,958
Legacy Sequoia | Business purpose residential loans, held-for-investment
Variable Interest Entity [Line Items]
Assets	0
Legacy Sequoia | Multifamily loans, held-for-investment
Variable Interest Entity [Line Items]
Assets	0	0
Legacy Sequoia | Other investments
Variable Interest Entity [Line Items]
Assets	0	0
Legacy Sequoia | Cash and cash equivalents
Variable Interest Entity [Line Items]
Assets	0
Legacy Sequoia | Restricted cash
Variable Interest Entity [Line Items]
Assets	143	146
Legacy Sequoia | Accrued interest receivable
Variable Interest Entity [Line Items]
Assets	655	822
Legacy Sequoia | Other assets
Variable Interest Entity [Line Items]
Assets	460	3,943
Legacy Sequoia | Short-term debt
Variable Interest Entity [Line Items]
Liabilities	0	0
Legacy Sequoia | Accrued interest payable
Variable Interest Entity [Line Items]
Liabilities	395	571
Legacy Sequoia | Accrued expenses and other liabilities
Variable Interest Entity [Line Items]
Liabilities	0	0
Legacy Sequoia | Asset-backed securities issued
Variable Interest Entity [Line Items]
Liabilities	402,465	512,240
Sequoia Choice
Variable Interest Entity [Line Items]
Assets	2,301,314	2,089,392
Liabilities	$ 2,044,957	$ 1,893,212
Number of VIEs | investment	9	6
Sequoia Choice | Residential loans, held-for-investment
Variable Interest Entity [Line Items]
Assets	$ 2,291,463	$ 2,079,382
Sequoia Choice | Business purpose residential loans, held-for-investment
Variable Interest Entity [Line Items]
Assets	0
Sequoia Choice | Multifamily loans, held-for-investment
Variable Interest Entity [Line Items]
Assets	0	0
Sequoia Choice | Other investments
Variable Interest Entity [Line Items]
Assets	0	0
Sequoia Choice | Cash and cash equivalents
Variable Interest Entity [Line Items]
Assets	0
Sequoia Choice | Restricted cash
Variable Interest Entity [Line Items]
Assets	27	1,022
Sequoia Choice | Accrued interest receivable
Variable Interest Entity [Line Items]
Assets	9,824	8,988
Sequoia Choice | Other assets
Variable Interest Entity [Line Items]
Assets	0	0
Sequoia Choice | Short-term debt
Variable Interest Entity [Line Items]
Liabilities	0	0
Sequoia Choice | Accrued interest payable
Variable Interest Entity [Line Items]
Liabilities	7,732	7,180
Sequoia Choice | Accrued expenses and other liabilities
Variable Interest Entity [Line Items]
Liabilities	27	1,022
Sequoia Choice | Asset-backed securities issued
Variable Interest Entity [Line Items]
Liabilities	2,037,198	1,885,010
Freddie Mac SLST
Variable Interest Entity [Line Items]
Assets	2,374,973	1,226,595
Liabilities	$ 1,923,696	$ 996,655
Number of VIEs | investment	2	1
Freddie Mac SLST | Residential loans, held-for-investment
Variable Interest Entity [Line Items]
Assets	$ 2,367,215	$ 1,222,669
Freddie Mac SLST | Business purpose residential loans, held-for-investment
Variable Interest Entity [Line Items]
Assets	0
Freddie Mac SLST | Multifamily loans, held-for-investment
Variable Interest Entity [Line Items]
Assets	0	0
Freddie Mac SLST | Other investments
Variable Interest Entity [Line Items]
Assets	0	0
Freddie Mac SLST | Cash and cash equivalents
Variable Interest Entity [Line Items]
Assets	0
Freddie Mac SLST | Restricted cash
Variable Interest Entity [Line Items]
Assets	0	0
Freddie Mac SLST | Accrued interest receivable
Variable Interest Entity [Line Items]
Assets	7,313	3,926
Freddie Mac SLST | Other assets
Variable Interest Entity [Line Items]
Assets	445	0
Freddie Mac SLST | Short-term debt
Variable Interest Entity [Line Items]
Liabilities	0	0
Freddie Mac SLST | Accrued interest payable
Variable Interest Entity [Line Items]
Liabilities	5,374	2,907
Freddie Mac SLST | Accrued expenses and other liabilities
Variable Interest Entity [Line Items]
Liabilities	0	0
Freddie Mac SLST | Asset-backed securities issued
Variable Interest Entity [Line Items]
Liabilities	1,918,322	993,748
Freddie Mac K-Series
Variable Interest Entity [Line Items]
Assets	4,422,063	2,151,193
Liabilities	$ 4,169,126	$ 2,025,314
Number of VIEs | investment	5	3
Freddie Mac K-Series | Residential loans, held-for-investment
Variable Interest Entity [Line Items]
Assets	$ 0	$ 0
Freddie Mac K-Series | Business purpose residential loans, held-for-investment
Variable Interest Entity [Line Items]
Assets	0
Freddie Mac K-Series | Multifamily loans, held-for-investment
Variable Interest Entity [Line Items]
Assets	4,408,524	2,144,598
Freddie Mac K-Series | Other investments
Variable Interest Entity [Line Items]
Assets	0	0
Freddie Mac K-Series | Cash and cash equivalents
Variable Interest Entity [Line Items]
Assets	0
Freddie Mac K-Series | Restricted cash
Variable Interest Entity [Line Items]
Assets	0	0
Freddie Mac K-Series | Accrued interest receivable
Variable Interest Entity [Line Items]
Assets	13,539	6,595
Freddie Mac K-Series | Other assets
Variable Interest Entity [Line Items]
Assets	0	0
Freddie Mac K-Series | Short-term debt
Variable Interest Entity [Line Items]
Liabilities	0	0
Freddie Mac K-Series | Accrued interest payable
Variable Interest Entity [Line Items]
Liabilities	12,887	6,239
Freddie Mac K-Series | Accrued expenses and other liabilities
Variable Interest Entity [Line Items]
Liabilities	0	0
Freddie Mac K-Series | Asset-backed securities issued
Variable Interest Entity [Line Items]
Liabilities	4,156,239	2,019,075
CAFL
Variable Interest Entity [Line Items]
Assets	2,203,919	0
Liabilities	$ 2,008,736	$ 0
Number of VIEs | investment	10	0
CAFL | Residential loans, held-for-investment
Variable Interest Entity [Line Items]
Assets	$ 0	$ 0
CAFL | Business purpose residential loans, held-for-investment
Variable Interest Entity [Line Items]
Assets	2,192,552
CAFL | Multifamily loans, held-for-investment
Variable Interest Entity [Line Items]
Assets	0	0
CAFL | Other investments
Variable Interest Entity [Line Items]
Assets	0	0
CAFL | Cash and cash equivalents
Variable Interest Entity [Line Items]
Assets	0
CAFL | Restricted cash
Variable Interest Entity [Line Items]
Assets	0	0
CAFL | Accrued interest receivable
Variable Interest Entity [Line Items]
Assets	9,572	0
CAFL | Other assets
Variable Interest Entity [Line Items]
Assets	1,795	0
CAFL | Short-term debt
Variable Interest Entity [Line Items]
Liabilities	0	0
CAFL | Accrued interest payable
Variable Interest Entity [Line Items]
Liabilities	7,485	0
CAFL | Accrued expenses and other liabilities
Variable Interest Entity [Line Items]
Liabilities	0	0
CAFL | Asset-backed securities issued
Variable Interest Entity [Line Items]
Liabilities	2,001,251	0
Servicing Investment
Variable Interest Entity [Line Items]
Assets	220,452	339,142
Liabilities	$ 167,697	$ 281,815
Number of VIEs | investment	3	3
Servicing Investment | Residential loans, held-for-investment
Variable Interest Entity [Line Items]
Assets	$ 0	$ 0
Servicing Investment | Business purpose residential loans, held-for-investment
Variable Interest Entity [Line Items]
Assets	0
Servicing Investment | Multifamily loans, held-for-investment
Variable Interest Entity [Line Items]
Assets	0	0
Servicing Investment | Other investments
Variable Interest Entity [Line Items]
Assets	184,802	312,688
Servicing Investment | Cash and cash equivalents
Variable Interest Entity [Line Items]
Assets	9,015
Servicing Investment | Restricted cash
Variable Interest Entity [Line Items]
Assets	21,766	25,363
Servicing Investment | Accrued interest receivable
Variable Interest Entity [Line Items]
Assets	4,869	1,091
Servicing Investment | Other assets
Variable Interest Entity [Line Items]
Assets	0	0
Servicing Investment | Short-term debt
Variable Interest Entity [Line Items]
Liabilities	152,554	262,740
Servicing Investment | Accrued interest payable
Variable Interest Entity [Line Items]
Liabilities	187	483
Servicing Investment | Accrued expenses and other liabilities
Variable Interest Entity [Line Items]
Liabilities	14,956	18,592
Servicing Investment | Asset-backed securities issued
Variable Interest Entity [Line Items]
Liabilities	$ 0	$ 0