Principles of Consolidation - Income (Loss) from Consolidated VIEs Accounted for as Collateralized Financing Entities (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2020	Sep. 30, 2019	Sep. 30, 2020	Sep. 30, 2019
Variable Interest Entity [Line Items]
Interest income	$ 121,982	$ 150,117	$ 450,009	$ 429,700
Interest expense	(100,411)	(116,604)	(349,748)	(332,100)
Net Interest Income	21,571	33,513	100,261	97,600
Investment fair value changes, net	107,047	11,444	(611,557)	34,741
Total non-interest income (loss), net	166,930	30,029	(552,648)	107,792
General and administrative expenses	(27,630)	(24,899)	(84,832)	(70,722)
Net Income (Loss)	141,812	34,310	(636,142)	120,040
Variable Interest Entity, Primary Beneficiary
Variable Interest Entity [Line Items]
Interest income	89,912	84,431	303,265	234,171
Interest expense	(67,871)	(73,956)	(238,763)	(206,378)
Net Interest Income	22,041	10,475	64,502	27,793
Investment fair value changes, net	101,420	28,023	(189,448)	56,936
Total non-interest income (loss), net	101,420	28,023	(189,448)	56,936
General and administrative expenses	(41)	(16)	(784)	(87)
Other expenses	(471)	(395)	346	(864)
Net Income (Loss)	122,949	38,087	(125,384)	83,778
Legacy Sequoia | Variable Interest Entity, Primary Beneficiary
Variable Interest Entity [Line Items]
Interest income	1,795	4,295	7,675	13,924
Interest expense	(1,058)	(3,452)	(5,098)	(11,548)
Net Interest Income	737	843	2,577	2,376
Investment fair value changes, net	(81)	(407)	(702)	(904)
Total non-interest income (loss), net	(81)	(407)	(702)	(904)
General and administrative expenses	0	0	0	0
Other expenses	0	0	0	0
Net Income (Loss)	656	436	1,875	1,472
Sequoia Choice | Variable Interest Entity, Primary Beneficiary
Variable Interest Entity [Line Items]
Interest income	20,919	27,555	68,566	80,045
Interest expense	(17,828)	(23,576)	(58,455)	(68,823)
Net Interest Income	3,091	3,979	10,111	11,222
Investment fair value changes, net	7,851	2,722	(22,065)	8,866
Total non-interest income (loss), net	7,851	2,722	(22,065)	8,866
General and administrative expenses	0	0	0	0
Other expenses	0	0	0	0
Net Income (Loss)	10,942	6,701	(11,954)	20,088
Freddie Mac SLST | Variable Interest Entity, Primary Beneficiary
Variable Interest Entity [Line Items]
Interest income	21,696	11,830	64,869	35,221
Interest expense	(15,473)	(8,709)	(47,495)	(26,013)
Net Interest Income	6,223	3,121	17,374	9,208
Investment fair value changes, net	82,214	17,300	(33,081)	31,702
Total non-interest income (loss), net	82,214	17,300	(33,081)	31,702
General and administrative expenses	0	0	0	0
Other expenses	0	0	0	0
Net Income (Loss)	88,437	20,421	(15,707)	40,910
Freddie Mac K-Series | Variable Interest Entity, Primary Beneficiary
Variable Interest Entity [Line Items]
Interest income	4,918	36,829	49,960	94,134
Interest expense	(4,426)	(35,328)	(47,154)	(90,089)
Net Interest Income	492	1,501	2,806	4,045
Investment fair value changes, net	2,166	7,445	(82,744)	13,810
Total non-interest income (loss), net	2,166	7,445	(82,744)	13,810
General and administrative expenses	0	0	0	0
Other expenses	0	0	0	0
Net Income (Loss)	2,658	8,946	(79,938)	17,855
CAFL | Variable Interest Entity, Primary Beneficiary
Variable Interest Entity [Line Items]
Interest income	36,181	0	99,169	0
Interest expense	(27,499)	0	(75,600)	0
Net Interest Income	8,682	0	23,569	0
Investment fair value changes, net	9,692	0	(41,841)	0
Total non-interest income (loss), net	9,692	0	(41,841)	0
General and administrative expenses	0	0	0	0
Other expenses	0	0	0	0
Net Income (Loss)	18,374	0	(18,272)	0
Servicing Investment | Variable Interest Entity, Primary Beneficiary
Variable Interest Entity [Line Items]
Interest income	4,403	3,922	13,026	10,847
Interest expense	(1,587)	(2,891)	(4,961)	(9,905)
Net Interest Income	2,816	1,031	8,065	942
Investment fair value changes, net	(422)	963	(9,015)	3,462
Total non-interest income (loss), net	(422)	963	(9,015)	3,462
General and administrative expenses	(41)	(16)	(784)	(87)
Other expenses	(471)	(395)	346	(864)
Net Income (Loss)	$ 1,882	$ 1,583	$ (1,388)	$ 3,453