Segment Information - Financial Information by Segment (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2020	Mar. 31, 2019
Segment Reporting Information [Line Items]
Interest income	$ 198,081	$ 131,041
Interest expense	(146,671)	(99,276)
Net Interest Income	51,410	31,765
Non-interest (Loss) Income
Mortgage banking activities, net	(28,411)	12,309
Investment fair value changes, net	(870,832)	20,159
Other income	2,437	4,625
Realized gains, net	3,852	10,686
Total non-interest (loss) income, net	(892,954)	47,779
General and administrative expenses	(32,668)	(23,159)
Other expenses	(91,415)	(1,038)
Benefit from (provision for) income taxes	22,229	(883)
Net (Loss) Income	(943,398)	54,464
Non-cash amortization income (expense), net	(4,497)	345
Other significant non-cash expense: goodwill impairment	(88,675)	0
Operating Segments | Residential Lending
Segment Reporting Information [Line Items]
Interest income	60,631	66,839
Interest expense	(41,402)	(47,676)
Net Interest Income	19,229	19,163
Non-interest (Loss) Income
Mortgage banking activities, net	(23,081)	10,827
Investment fair value changes, net	(196,635)	(1,720)
Other income	(497)	1,449
Realized gains, net	1,796	4,937
Total non-interest (loss) income, net	(218,417)	15,493
General and administrative expenses	(5,632)	(7,203)
Other expenses	0	0
Benefit from (provision for) income taxes	5,330	(501)
Net (Loss) Income	(199,490)	26,952
Non-cash amortization income (expense), net	367	1,975
Other significant non-cash expense: goodwill impairment	0
Operating Segments | Business Purpose Lending
Segment Reporting Information [Line Items]
Interest income	53,060	2,937
Interest expense	(34,990)	(1,539)
Net Interest Income	18,070	1,398
Non-interest (Loss) Income
Mortgage banking activities, net	(5,330)	1,482
Investment fair value changes, net	(142,130)	(303)
Other income	1,693	466
Realized gains, net	0	0
Total non-interest (loss) income, net	(145,767)	1,645
General and administrative expenses	(14,333)	(2,565)
Other expenses	(92,985)	(633)
Benefit from (provision for) income taxes	6,582	(5)
Net (Loss) Income	(228,433)	(160)
Non-cash amortization income (expense), net	(5,363)	(732)
Other significant non-cash expense: goodwill impairment	(88,675)
Operating Segments | Multifamily Investments
Segment Reporting Information [Line Items]
Interest income	46,883	28,208
Interest expense	(43,286)	(25,870)
Net Interest Income	3,597	2,338
Non-interest (Loss) Income
Mortgage banking activities, net	0	0
Investment fair value changes, net	(227,122)	8,524
Other income	1,240	0
Realized gains, net	(1,604)	0
Total non-interest (loss) income, net	(227,486)	8,524
General and administrative expenses	(610)	(323)
Other expenses	0	0
Benefit from (provision for) income taxes	(106)	0
Net (Loss) Income	(224,605)	10,539
Non-cash amortization income (expense), net	54	(136)
Other significant non-cash expense: goodwill impairment	0
Operating Segments | Third-Party Residential Investments
Segment Reporting Information [Line Items]
Interest income	34,313	28,204
Interest expense	(24,471)	(20,076)
Net Interest Income	9,842	8,128
Non-interest (Loss) Income
Mortgage banking activities, net	0	0
Investment fair value changes, net	(304,436)	14,055
Other income	1	0
Realized gains, net	3,660	5,749
Total non-interest (loss) income, net	(300,775)	19,804
General and administrative expenses	(1,178)	(663)
Other expenses	1,882	(227)
Benefit from (provision for) income taxes	10,423	(377)
Net (Loss) Income	(279,806)	26,665
Non-cash amortization income (expense), net	812	(271)
Other significant non-cash expense: goodwill impairment	0
Corporate/ Other
Segment Reporting Information [Line Items]
Interest income	3,194	4,853
Interest expense	(2,522)	(4,115)
Net Interest Income	672	738
Non-interest (Loss) Income
Mortgage banking activities, net	0	0
Investment fair value changes, net	(509)	(397)
Other income	0	2,710
Realized gains, net	0	0
Total non-interest (loss) income, net	(509)	2,313
General and administrative expenses	(10,915)	(12,405)
Other expenses	(312)	(178)
Benefit from (provision for) income taxes	0	0
Net (Loss) Income	(11,064)	(9,532)
Non-cash amortization income (expense), net	(367)	$ (491)
Other significant non-cash expense: goodwill impairment	$ 0