Principles of Consolidation - Income (Loss) from Consolidated VIEs (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2020	Jun. 30, 2019	Jun. 30, 2020	Jun. 30, 2019
Variable Interest Entity [Line Items]
Interest income	$ 129,946	$ 148,542	$ 328,027	$ 279,583
Interest expense	(102,666)	(116,220)	(249,337)	(215,496)
Net Interest Income	27,280	32,322	78,690	64,087
Investment fair value changes, net	152,228	3,138	(718,604)	23,297
Total non-interest income (loss), net	173,376	29,984	(719,578)	77,763
General and administrative expenses	(30,092)	(26,255)	(62,760)	(49,414)
Net Income (Loss)	165,444	31,266	(777,954)	85,730
Variable Interest Entity, Primary Beneficiary
Variable Interest Entity [Line Items]
Interest income	88,826	82,697	213,353	149,741
Interest expense	(67,101)	(73,514)	(170,892)	(132,421)
Net Interest Income	21,725	9,183	42,461	17,320
Investment fair value changes, net	87,594	15,108	(290,868)	28,913
Total non-interest income (loss), net	87,594	15,108	(290,868)	28,913
General and administrative expenses	(712)	(41)	(743)	(70)
Other expenses	(1,065)	(242)	817	(468)
Net Income (Loss)	107,542	24,008	(248,333)	45,695
Legacy Sequoia | Variable Interest Entity, Primary Beneficiary
Variable Interest Entity [Line Items]
Interest income	2,686	4,776	5,880	9,629
Interest expense	(1,518)	(3,981)	(4,040)	(8,096)
Net Interest Income	1,168	795	1,840	1,533
Investment fair value changes, net	(230)	(123)	(621)	(497)
Total non-interest income (loss), net	(230)	(123)	(621)	(497)
General and administrative expenses	0	0	0	0
Other expenses	0	0	0	0
Net Income (Loss)	938	672	1,219	1,036
Sequoia Choice | Variable Interest Entity, Primary Beneficiary
Variable Interest Entity [Line Items]
Interest income	22,565	26,828	47,647	52,490
Interest expense	(19,117)	(23,134)	(40,627)	(45,247)
Net Interest Income	3,448	3,694	7,020	7,243
Investment fair value changes, net	39,753	2,879	(29,916)	6,144
Total non-interest income (loss), net	39,753	2,879	(29,916)	6,144
General and administrative expenses	0	0	0	0
Other expenses	0	0	0	0
Net Income (Loss)	43,201	6,573	(22,896)	13,387
Freddie Mac SLST | Variable Interest Entity, Primary Beneficiary
Variable Interest Entity [Line Items]
Interest income	21,187	11,597	43,173	23,391
Interest expense	(15,845)	(8,557)	(32,022)	(17,304)
Net Interest Income	5,342	3,040	11,151	6,087
Investment fair value changes, net	26,867	8,037	(115,295)	14,402
Total non-interest income (loss), net	26,867	8,037	(115,295)	14,402
General and administrative expenses	0	0	0	0
Other expenses	0	0	0	0
Net Income (Loss)	32,209	11,077	(104,144)	20,489
Freddie Mac K-Series | Variable Interest Entity, Primary Beneficiary
Variable Interest Entity [Line Items]
Interest income	4,870	35,917	45,042	57,305
Interest expense	(4,378)	(34,441)	(42,728)	(54,760)
Net Interest Income	492	1,476	2,314	2,545
Investment fair value changes, net	1,599	3,246	(84,910)	6,365
Total non-interest income (loss), net	1,599	3,246	(84,910)	6,365
General and administrative expenses	0	0	0	0
Other expenses	0	0	0	0
Net Income (Loss)	2,091	4,722	(82,596)	8,910
CAFL | Variable Interest Entity, Primary Beneficiary
Variable Interest Entity [Line Items]
Interest income	32,978	0	62,988	0
Interest expense	(24,446)	0	(48,101)	0
Net Interest Income	8,532	0	14,887	0
Investment fair value changes, net	16,313	0	(51,533)	0
Total non-interest income (loss), net	16,313	0	(51,533)	0
General and administrative expenses	0	0	0	0
Other expenses	0	0	0	0
Net Income (Loss)	24,845	0	(36,646)	0
Servicing Investment | Variable Interest Entity, Primary Beneficiary
Variable Interest Entity [Line Items]
Interest income	4,540	3,579	8,623	6,926
Interest expense	(1,797)	(3,401)	(3,374)	(7,014)
Net Interest Income	2,743	178	5,249	(88)
Investment fair value changes, net	3,292	1,069	(8,593)	2,499
Total non-interest income (loss), net	3,292	1,069	(8,593)	2,499
General and administrative expenses	(712)	(41)	(743)	(70)
Other expenses	(1,065)	(242)	817	(468)
Net Income (Loss)	$ 4,258	$ 964	$ (3,270)	$ 1,873