Segment Information - Financial Information by Segment (Details) - USD ($) $ in Thousands	3 Months Ended			9 Months Ended
	Sep. 30, 2020	Mar. 31, 2020	Sep. 30, 2019	Sep. 30, 2020	Sep. 30, 2019
Segment Reporting Information [Line Items]
Interest income	$ 121,982		$ 150,117	$ 450,009	$ 429,700
Interest expense	(100,411)		(116,604)	(349,748)	(332,100)
Net Interest Income	21,571		33,513	100,261	97,600
Non-interest Income (Loss)
Mortgage banking activities, net	59,395		9,515	24,511	40,984
Investment fair value changes, net	107,047		11,444	(611,557)	34,741
Other income, net	(114)		4,356	3,979	13,840
Realized gains, net	602		4,714	30,419	18,227
Total non-interest income (loss), net	166,930		30,029	(552,648)	107,792
General and administrative expenses	(27,630)		(24,899)	(84,832)	(70,722)
Loan acquisition costs	(2,158)		(1,916)	(7,716)	(5,507)
Other expenses	(7,788)		(2,531)	(104,286)	(6,021)
(Provision for) benefit from income taxes	(9,113)		114	13,079	(3,102)
Net Income (Loss)	141,812		34,310	(636,142)	120,040
Non-cash amortization (expense) income, net	(6,333)		(718)	(19,378)	(890)
Other significant non-cash expense: goodwill impairment	0	$ (89,000)	0	(88,675)	0
Operating Segments | Residential Lending
Segment Reporting Information [Line Items]
Interest income	26,672		69,586	123,956	204,036
Interest expense	(22,541)		(48,798)	(94,112)	(145,795)
Net Interest Income	4,131		20,788	29,844	58,241
Non-interest Income (Loss)
Mortgage banking activities, net	11,864		5,016	(19,222)	31,202
Investment fair value changes, net	2,443		(11,088)	(159,107)	(20,910)
Other income, net	(2,011)		1,878	(2,278)	6,317
Realized gains, net	0		0	2,001	7,728
Total non-interest income (loss), net	12,296		(4,194)	(178,606)	24,337
General and administrative expenses	(4,602)		(5,581)	(12,901)	(17,591)
Loan acquisition costs	(304)		(867)	(1,512)	(2,859)
Other expenses	(3,309)		0	(3,309)	0
(Provision for) benefit from income taxes	(826)		228	7,827	(1,757)
Net Income (Loss)	7,386		10,374	(158,657)	60,371
Non-cash amortization (expense) income, net	1,785		2,537	602	6,780
Other significant non-cash expense: goodwill impairment				0
Operating Segments | Business Purpose Lending
Segment Reporting Information [Line Items]
Interest income	55,930		5,527	162,732	12,595
Interest expense	(46,027)		(2,813)	(119,854)	(6,590)
Net Interest Income	9,903		2,714	42,878	6,005
Non-interest Income (Loss)
Mortgage banking activities, net	47,531		4,499	43,733	9,782
Investment fair value changes, net	16,892		(1,073)	(84,837)	(1,833)
Other income, net	623		1,918	3,493	4,213
Realized gains, net	0		0	0	0
Total non-interest income (loss), net	65,046		5,344	(37,611)	12,162
General and administrative expenses	(9,321)		(6,028)	(29,977)	(13,452)
Loan acquisition costs	(1,660)		(756)	(5,630)	(2,017)
Other expenses	(3,874)		(1,900)	(100,743)	(4,432)
(Provision for) benefit from income taxes	(8,544)		(8)	477	25
Net Income (Loss)	51,550		(634)	(130,606)	(1,709)
Non-cash amortization (expense) income, net	(6,719)		(2,107)	(18,050)	(4,907)
Other significant non-cash expense: goodwill impairment				(88,675)
Operating Segments | Third-Party Investments
Segment Reporting Information [Line Items]
Interest income	37,576		70,709	155,583	199,145
Interest expense	(27,534)		(61,541)	(123,160)	(168,167)
Net Interest Income	10,042		9,168	32,423	30,978
Non-interest Income (Loss)
Mortgage banking activities, net	0		0	0	0
Investment fair value changes, net	87,890		24,057	(366,696)	58,492
Other income, net	340		560	1,072	600
Realized gains, net	602		4,714	3,236	10,499
Total non-interest income (loss), net	88,832		29,331	(362,388)	69,591
General and administrative expenses	(709)		(676)	(4,230)	(2,239)
Loan acquisition costs	(194)		(180)	(567)	(503)
Other expenses	(470)		(395)	347	(864)
(Provision for) benefit from income taxes	257		(106)	4,775	(1,370)
Net Income (Loss)	97,758		37,142	(329,640)	95,593
Non-cash amortization (expense) income, net	117		0	1,105	(824)
Other significant non-cash expense: goodwill impairment				0
Corporate/ Other
Segment Reporting Information [Line Items]
Interest income	1,804		4,295	7,738	13,924
Interest expense	(4,309)		(3,452)	(12,622)	(11,548)
Net Interest Income	(2,505)		843	(4,884)	2,376
Non-interest Income (Loss)
Mortgage banking activities, net	0		0	0	0
Investment fair value changes, net	(178)		(452)	(917)	(1,008)
Other income, net	934		0	1,692	2,710
Realized gains, net	0		0	25,182	0
Total non-interest income (loss), net	756		(452)	25,957	1,702
General and administrative expenses	(12,998)		(12,614)	(37,724)	(37,440)
Loan acquisition costs	0		(113)	(7)	(128)
Other expenses	(135)		(236)	(581)	(725)
(Provision for) benefit from income taxes	0		0	0	0
Net Income (Loss)	(14,882)		(12,572)	(17,239)	(34,215)
Non-cash amortization (expense) income, net	$ (1,516)		$ (1,148)	(3,035)	$ (1,939)
Other significant non-cash expense: goodwill impairment				$ 0