Principles of Consolidation - Income (Loss) from Consolidated VIEs Accounted for as Collateralized Financing Entities (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2020	Dec. 31, 2019	Dec. 31, 2018
Variable Interest Entity [Line Items]
Interest income	$ 571,916	$ 622,281	$ 378,717
Interest expense	(448,005)	(479,808)	(239,039)
Net Interest Income	123,911	142,473	139,678
Investment fair value changes, net	(588,438)	35,500	(25,689)
Total non-interest income, net	(475,354)	165,844	73,988
General and administrative expenses	(115,204)	(108,737)	(75,298)
Net (Loss) Income	(581,847)	169,183	$ 119,600
Variable Interest Entity, Primary Beneficiary
Variable Interest Entity [Line Items]
Interest income	391,191	354,470
Interest expense	(305,765)	(306,419)
Net Interest Income	85,426	48,051
Investment fair value changes, net	(167,856)	53,713
Total non-interest income, net	(167,856)	53,713
General and administrative expenses	(867)	(343)
Other expenses	193	(1,106)
Net (Loss) Income	(83,104)	100,315
Legacy Sequoia | Variable Interest Entity, Primary Beneficiary
Variable Interest Entity [Line Items]
Interest income	9,061	17,649
Interest expense	(5,945)	(14,418)
Net Interest Income	3,116	3,231
Investment fair value changes, net	(1,512)	(1,545)
Total non-interest income, net	(1,512)	(1,545)
General and administrative expenses	0	0
Other expenses	0	0
Net (Loss) Income	1,604	1,686
Sequoia Choice | Variable Interest Entity, Primary Beneficiary
Variable Interest Entity [Line Items]
Interest income	87,093	108,798
Interest expense	(73,643)	(93,354)
Net Interest Income	13,450	15,444
Investment fair value changes, net	(13,244)	6,947
Total non-interest income, net	(13,244)	6,947
General and administrative expenses	0	0
Other expenses	0	0
Net (Loss) Income	206	22,391
Freddie Mac SLST | Variable Interest Entity, Primary Beneficiary
Variable Interest Entity [Line Items]
Interest income	85,609	57,840
Interest expense	(62,483)	(42,574)
Net Interest Income	23,126	15,266
Investment fair value changes, net	(21,160)	27,206
Total non-interest income, net	(21,160)	27,206
General and administrative expenses	0	0
Other expenses	0	0
Net (Loss) Income	1,966	42,472
Freddie Mac K-Series | Variable Interest Entity, Primary Beneficiary
Variable Interest Entity [Line Items]
Interest income	54,813	132,600
Interest expense	(51,521)	(126,948)
Net Interest Income	3,292	5,652
Investment fair value changes, net	(81,039)	21,430
Total non-interest income, net	(81,039)	21,430
General and administrative expenses	0	0
Other expenses	0	0
Net (Loss) Income	(77,747)	27,082
CAFL | Variable Interest Entity, Primary Beneficiary
Variable Interest Entity [Line Items]
Interest income	136,950	23,072
Interest expense	(105,732)	(17,173)
Net Interest Income	31,218	5,899
Investment fair value changes, net	(39,574)	(3,636)
Total non-interest income, net	(39,574)	(3,636)
General and administrative expenses	0	0
Other expenses	0	0
Net (Loss) Income	(8,356)	2,263
Servicing Investment | Variable Interest Entity, Primary Beneficiary
Variable Interest Entity [Line Items]
Interest income	17,665	14,511
Interest expense	(6,441)	(11,952)
Net Interest Income	11,224	2,559
Investment fair value changes, net	(11,327)	3,311
Total non-interest income, net	(11,327)	3,311
General and administrative expenses	(867)	(343)
Other expenses	193	(1,106)
Net (Loss) Income	$ (777)	$ 4,421