Segment Information - Financial Information by Segment (Details) - USD ($) $ in Thousands	3 Months Ended	12 Months Ended
	Mar. 31, 2020	Dec. 31, 2020	Dec. 31, 2019	Dec. 31, 2018
Segment Reporting Information [Line Items]
Interest income		$ 571,916	$ 622,281	$ 378,717
Interest expense		(448,005)	(479,808)	(239,039)
Net Interest Income		123,911	142,473	139,678
Non-interest income
Mortgage banking activities, net		78,472	87,266	59,566
Investment fair value changes, net		(588,438)	35,500	(25,689)
Other income		4,188	19,257	13,070
Realized gains, net		30,424	23,821	27,041
Total non-interest income, net		(475,354)	165,844	73,988
General and administrative expenses		(115,204)	(108,737)	(75,298)
Loan acquisition costs		(11,023)	(9,935)	(7,484)
Other expenses		(108,785)	(13,022)	(196)
Provision for income taxes		4,608	(7,440)	(11,088)
Net (Loss) Income		(581,847)	169,183	119,600
Non-cash amortization (expense) income, net		(25,324)	(3,361)	12,379
Other significant non-cash expense: goodwill impairment	$ (89,000)	(88,675)	0	0
Operating Segments | Residential Lending
Segment Reporting Information [Line Items]
Interest income		150,906	268,559	245,124
Interest expense		(107,371)	(171,119)	(134,590)
Net Interest Income		43,535	97,440	110,534
Non-interest income
Mortgage banking activities, net		3,721	47,743	59,623
Investment fair value changes, net		(153,388)	(27,920)	(21,686)
Other income		(4,642)	9,210	12,452
Realized gains, net		2,001	8,292	7,709
Total non-interest income, net		(152,308)	37,325	58,098
General and administrative expenses		(17,939)	(26,717)	(26,897)
Loan acquisition costs		(2,785)	(3,954)	(5,242)
Other expenses		(4,114)	0	0
Provision for income taxes		4,567	(4,074)	(8,033)
Net (Loss) Income		(129,044)	100,020	128,460
Non-cash amortization (expense) income, net		2,401	3,669	4,486
Other significant non-cash expense: goodwill impairment		0
Operating Segments | Business Purpose Lending
Segment Reporting Information [Line Items]
Interest income		218,890	54,372	4,588
Interest expense		(157,292)	(32,232)	(1,598)
Net Interest Income		61,598	22,140	2,990
Non-interest income
Mortgage banking activities, net		74,751	39,523	(57)
Investment fair value changes, net		(81,042)	(6,722)	(29)
Other income		4,651	5,852	0
Realized gains, net		0	0	0
Total non-interest income, net		(1,640)	38,653	(86)
General and administrative expenses		(39,319)	(25,591)	(1,948)
Loan acquisition costs		(7,544)	(5,064)	(649)
Other expenses		(104,147)	(8,521)	0
Provision for income taxes		(4,063)	(947)	0
Net (Loss) Income		(95,115)	20,670	307
Non-cash amortization (expense) income, net		(24,638)	(9,173)	(290)
Other significant non-cash expense: goodwill impairment		(88,675)
Operating Segments | Third-Party Investments
Segment Reporting Information [Line Items]
Interest income		192,984	281,701	108,969
Interest expense		(135,722)	(213,312)	(41,887)
Net Interest Income		57,262	68,389	67,082
Non-interest income
Mortgage banking activities, net		0	0	0
Investment fair value changes, net		(352,004)	71,759	(2,978)
Other income		1,494	1,484	0
Realized gains, net		3,241	15,529	19,332
Total non-interest income, net		(347,269)	88,772	16,354
General and administrative expenses		(5,046)	(3,561)	(2,140)
Loan acquisition costs		(684)	(780)	(1,584)
Other expenses		194	(1,106)	(18)
Provision for income taxes		4,104	(2,419)	(3,055)
Net (Loss) Income		(291,439)	149,295	76,639
Non-cash amortization (expense) income, net		1,867	6,956	12,294
Other significant non-cash expense: goodwill impairment		0
Corporate/ Other
Segment Reporting Information [Line Items]
Interest income		9,136	17,649	20,036
Interest expense		(47,620)	(63,145)	(60,964)
Net Interest Income		(38,484)	(45,496)	(40,928)
Non-interest income
Mortgage banking activities, net		0	0	0
Investment fair value changes, net		(2,004)	(1,617)	(996)
Other income		2,685	2,711	618
Realized gains, net		25,182	0	0
Total non-interest income, net		25,863	1,094	(378)
General and administrative expenses		(52,900)	(52,868)	(44,313)
Loan acquisition costs		(10)	(137)	(9)
Other expenses		(718)	(3,395)	(178)
Provision for income taxes		0	0	0
Net (Loss) Income		(66,249)	(100,802)	(85,806)
Non-cash amortization (expense) income, net		(4,954)	$ (4,813)	$ (4,111)
Other significant non-cash expense: goodwill impairment		$ 0