Segment Information - Financial Information by Segment (Details) - USD ($) $ in Thousands	3 Months Ended	12 Months Ended
	Mar. 31, 2020	Dec. 31, 2022	Dec. 31, 2021	Dec. 31, 2020
Segment Reporting Information [Line Items]
Interest income		$ 707,854	$ 574,926	$ 571,916
Interest expense		(552,400)	(426,749)	(448,005)
Net Interest Income		155,454	148,177	123,911
Non-interest (loss) income
Mortgage banking activities, net		(13,659)	235,744	78,472
Investment fair value changes, net		(175,558)	128,049	(588,438)
Other income, net		21,204	12,018	4,188
Realized gains, net		5,334	17,993	30,424
Total non-interest (loss) income, net		(162,679)	393,804	(475,354)
General and administrative expenses		(140,908)	(165,218)	(113,498)
Portfolio management costs		(7,951)	(5,758)	(4,204)
Loan acquisition costs		(11,766)	(16,219)	(8,525)
Other expense				(108,785)
Other expenses		(15,590)	(16,695)	(108,785)
Benefit from (provision for) income taxes		19,920	(18,478)	4,608
Net (Loss) Income		(163,520)	319,613	(581,847)
Non-cash amortization (expense) income, net		(21,928)	(45,193)	(25,324)
Other significant non-cash expense: goodwill impairment	$ (89,000)	0	0	(88,675)
Operating Segments | Residential Mortgage Banking
Segment Reporting Information [Line Items]
Interest income		45,202	48,953	17,839
Interest expense		(32,735)	(26,963)	(11,978)
Net Interest Income		12,467	21,990	5,861
Non-interest (loss) income
Mortgage banking activities, net		(21,282)	127,151	3,721
Investment fair value changes, net		0	0	0
Other income, net		0	0	0
Realized gains, net		0	0	0
Total non-interest (loss) income, net		(21,282)	127,151	3,721
General and administrative expenses		(22,566)	(33,574)	(16,318)
Portfolio management costs		0	0	(50)
Loan acquisition costs		(3,085)	(7,480)	(2,656)
Other expense				(4,114)
Other expenses		74	104
Benefit from (provision for) income taxes		12,814	(25,777)	4,567
Net (Loss) Income		(21,578)	82,414	(8,989)
Non-cash amortization (expense) income, net		(1,075)	(82)	(662)
Other significant non-cash expense: goodwill impairment				0
Operating Segments | Business Purpose Mortgage Banking
Segment Reporting Information [Line Items]
Interest income		28,674	14,054	19,200
Interest expense		(18,041)	(7,230)	(13,145)
Net Interest Income		10,633	6,824	6,055
Non-interest (loss) income
Mortgage banking activities, net		7,623	108,593	74,622
Investment fair value changes, net		0	0	(101)
Other income, net		3,509	1,046	3,228
Realized gains, net		0	0	0
Total non-interest (loss) income, net		11,132	109,639	77,749
General and administrative expenses		(56,557)	(46,586)	(37,461)
Portfolio management costs		0	0	0
Loan acquisition costs		(8,681)	(8,100)	(5,859)
Other expense				(104,147)
Other expenses		(13,969)	(15,127)
Benefit from (provision for) income taxes		13,157	(8,122)	(4,063)
Net (Loss) Income		(44,285)	38,528	(67,726)
Non-cash amortization (expense) income, net		(15,071)	(16,452)	(18,426)
Other significant non-cash expense: goodwill impairment				(88,675)
Operating Segments | Investment Portfolio
Segment Reporting Information [Line Items]
Interest income		627,134	507,173	525,741
Interest expense		(445,154)	(351,635)	(375,262)
Net Interest Income		181,980	155,538	150,479
Non-interest (loss) income
Mortgage banking activities, net		0	0	129
Investment fair value changes, net		(191,148)	129,614	(586,333)
Other income, net		18,596	10,021	(1,725)
Realized gains, net		3,174	17,993	5,242
Total non-interest (loss) income, net		(169,378)	157,628	(582,687)
General and administrative expenses		(6,036)	(7,992)	(6,819)
Portfolio management costs		(7,951)	(5,758)	(4,154)
Loan acquisition costs		0	(635)	0
Other expense				194
Other expenses		(1,695)	(1,689)
Benefit from (provision for) income taxes		(6,051)	(3,862)	4,104
Net (Loss) Income		(9,131)	293,230	(438,883)
Non-cash amortization (expense) income, net		2,507	(20,781)	(1,282)
Other significant non-cash expense: goodwill impairment				0
Corporate/ Other
Segment Reporting Information [Line Items]
Interest income		6,844	4,746	9,136
Interest expense		(56,470)	(40,921)	(47,620)
Net Interest Income		(49,626)	(36,175)	(38,484)
Non-interest (loss) income
Mortgage banking activities, net		0	0	0
Investment fair value changes, net		15,590	(1,565)	(2,004)
Other income, net		(901)	951	2,685
Realized gains, net		2,160	0	25,182
Total non-interest (loss) income, net		16,849	(614)	25,863
General and administrative expenses		(55,749)	(77,066)	(52,900)
Portfolio management costs		0	0	0
Loan acquisition costs		0	(4)	(10)
Other expense				(718)
Other expenses		0	17	(718)
Benefit from (provision for) income taxes		0	19,283	0
Net (Loss) Income		(88,526)	(94,559)	(66,249)
Non-cash amortization (expense) income, net		$ (8,289)	$ (7,878)	(4,954)
Other significant non-cash expense: goodwill impairment				$ 0