Segment Information - Financial Information by Segment (Details) - USD ($) $ in Thousands	3 Months Ended			6 Months Ended
	Jun. 30, 2020	Mar. 31, 2020	Jun. 30, 2019	Jun. 30, 2020	Jun. 30, 2019
Segment Reporting Information [Line Items]
Interest income	$ 129,946		$ 148,542	$ 328,027	$ 279,583
Interest expense	(102,666)		(116,220)	(249,337)	(215,496)
Net Interest Income	27,280		32,322	78,690	64,087
Non-interest Income (Loss)
Mortgage banking activities, net	(5,772)		19,160	(34,183)	31,469
Investment fair value changes, net	152,228		3,138	(718,604)	23,297
Other income, net	955		4,859	3,392	9,484
Realized gains, net	25,965		2,827	29,817	13,513
Total non-interest income (loss), net	173,376		29,984	(719,578)	77,763
General and administrative expenses	(30,092)		(26,255)	(62,760)	(49,414)
Other expenses	(5,083)		(2,452)	(96,498)	(3,490)
Benefit from (provision for) income taxes	(37)		(2,333)	22,192	(3,216)
Net Income (Loss)	165,444		31,266	(777,954)	85,730
Non-cash amortization (expense) income, net	(8,963)		(517)	(13,044)	(36)
Other significant non-cash expense: goodwill impairment	0	$ (89,000)	0	(88,675)	0
Operating Segments | Residential Lending
Segment Reporting Information [Line Items]
Interest income	36,653		67,611	97,284	134,450
Interest expense	(30,169)		(49,321)	(71,571)	(96,997)
Net Interest Income	6,484		18,290	25,713	37,453
Non-interest Income (Loss)
Mortgage banking activities, net	(8,005)		15,359	(31,086)	26,186
Investment fair value changes, net	35,085		(8,102)	(161,550)	(9,822)
Other income, net	230		2,990	(267)	4,439
Realized gains, net	205		2,791	2,001	7,728
Total non-interest income (loss), net	27,515		13,038	(190,902)	28,531
General and administrative expenses	(3,875)		(6,799)	(9,507)	(14,002)
Other expenses	0		0	0	0
Benefit from (provision for) income taxes	3,323		(1,484)	8,653	(1,985)
Net Income (Loss)	33,447		23,045	(166,043)	49,997
Non-cash amortization (expense) income, net	(1,265)		2,268	(1,053)	4,243
Other significant non-cash expense: goodwill impairment				0
Operating Segments | Business Purpose Lending
Segment Reporting Information [Line Items]
Interest income	53,742		4,131	106,802	7,068
Interest expense	(38,837)		(2,238)	(73,827)	(3,777)
Net Interest Income	14,905		1,893	32,975	3,291
Non-interest Income (Loss)
Mortgage banking activities, net	2,233		3,801	(3,097)	5,283
Investment fair value changes, net	40,401		(457)	(101,729)	(760)
Other income, net	476		1,829	2,169	2,295
Realized gains, net	0		0	0	0
Total non-interest income (loss), net	43,110		5,173	(102,657)	6,818
General and administrative expenses	(10,293)		(6,120)	(24,626)	(8,685)
Other expenses	(3,884)		(1,899)	(96,869)	(2,532)
Benefit from (provision for) income taxes	2,439		38	9,021	33
Net Income (Loss)	46,277		(915)	(182,156)	(1,075)
Non-cash amortization (expense) income, net	(6,391)		(2,068)	(11,316)	(2,800)
Other significant non-cash expense: goodwill impairment				(88,675)
Operating Segments | Third-Party Investments
Segment Reporting Information [Line Items]
Interest income	36,811		72,024	118,007	128,436
Interest expense	(27,869)		(60,680)	(95,626)	(106,626)
Net Interest Income	8,942		11,344	22,381	21,810
Non-interest Income (Loss)
Mortgage banking activities, net	0		0	0	0
Investment fair value changes, net	76,972		11,856	(454,586)	34,435
Other income, net	(509)		40	732	40
Realized gains, net	578		36	2,634	5,785
Total non-interest income (loss), net	77,041		11,932	(451,220)	40,260
General and administrative expenses	(2,106)		(900)	(3,894)	(1,886)
Other expenses	(1,065)		(242)	817	(469)
Benefit from (provision for) income taxes	(5,799)		(887)	4,518	(1,264)
Net Income (Loss)	77,013		21,247	(427,398)	58,451
Non-cash amortization (expense) income, net	312		(417)	1,053	(688)
Other significant non-cash expense: goodwill impairment				0
Corporate/ Other
Segment Reporting Information [Line Items]
Interest income	2,740		4,776	5,934	9,629
Interest expense	(5,791)		(3,981)	(8,313)	(8,096)
Net Interest Income	(3,051)		795	(2,379)	1,533
Non-interest Income (Loss)
Mortgage banking activities, net	0		0	0	0
Investment fair value changes, net	(230)		(159)	(739)	(556)
Other income, net	758		0	758	2,710
Realized gains, net	25,182		0	25,182	0
Total non-interest income (loss), net	25,710		(159)	25,201	2,154
General and administrative expenses	(13,818)		(12,436)	(24,733)	(24,841)
Other expenses	(134)		(311)	(446)	(489)
Benefit from (provision for) income taxes	0		0	0	0
Net Income (Loss)	8,707		(12,111)	(2,357)	(21,643)
Non-cash amortization (expense) income, net	$ (1,619)		$ (300)	(1,728)	$ (791)
Other significant non-cash expense: goodwill impairment				$ 0