Segment Information - Financial Information by Segment (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2022	Mar. 31, 2021
Segment Reporting Information [Line Items]
Interest income	$ 189,400	$ 128,305
Interest expense	(136,298)	(102,552)
Net Interest Income	53,102	25,753
Non-interest Income
Mortgage banking activities, net	16,315	82,607
Investment fair value changes, net	(6,120)	45,087
Other income, net	5,983	3,843
Realized gains, net	2,581	2,716
Total non-interest income, net	18,759	134,253
General and administrative expenses	(34,854)	(43,551)
Loan acquisition costs	(4,465)	(3,559)
Other expenses	(4,085)	(4,096)
Benefit from (provision for) income taxes	2,458	(11,543)
Net Income	30,915	97,257
Non-cash amortization (expense) income, net	1,481	(6,230)
Operating Segments | Residential Mortgage Banking
Segment Reporting Information [Line Items]
Interest income	12,967	7,579
Interest expense	(6,936)	(4,747)
Net Interest Income	6,031	2,832
Non-interest Income
Mortgage banking activities, net	7,935	61,435
Investment fair value changes, net	0	0
Other income, net	0	0
Realized gains, net	0	0
Total non-interest income, net	7,935	61,435
General and administrative expenses	(6,101)	(12,689)
Loan acquisition costs	(1,417)	(1,404)
Other expenses	0	(6)
Benefit from (provision for) income taxes	1,007	(9,486)
Net Income	7,455	40,682
Non-cash amortization (expense) income, net	104	(3)
Operating Segments | Business Purpose Mortgage Banking
Segment Reporting Information [Line Items]
Interest income	4,841	2,855
Interest expense	(2,568)	(1,599)
Net Interest Income	2,273	1,256
Non-interest Income
Mortgage banking activities, net	8,380	21,172
Investment fair value changes, net	0	0
Other income, net	575	122
Realized gains, net	0	0
Total non-interest income, net	8,955	21,294
General and administrative expenses	(10,472)	(10,194)
Loan acquisition costs	(3,048)	(1,492)
Other expenses	(3,534)	(3,777)
Benefit from (provision for) income taxes	3,281	(1,321)
Net Income	(2,545)	5,766
Non-cash amortization (expense) income, net	(3,890)	(4,098)
Operating Segments | Investment Portfolio
Segment Reporting Information [Line Items]
Interest income	170,572	116,509
Interest expense	(116,581)	(85,945)
Net Interest Income	53,991	30,564
Non-interest Income
Mortgage banking activities, net	0	0
Investment fair value changes, net	(5,406)	45,761
Other income, net	5,282	3,574
Realized gains, net	2,581	2,716
Total non-interest income, net	2,457	52,051
General and administrative expenses	(3,133)	(3,164)
Loan acquisition costs	0	(659)
Other expenses	(551)	(330)
Benefit from (provision for) income taxes	(1,830)	(736)
Net Income	50,934	77,726
Non-cash amortization (expense) income, net	7,300	(234)
Corporate/ Other
Segment Reporting Information [Line Items]
Interest income	1,020	1,362
Interest expense	(10,213)	(10,261)
Net Interest Income	(9,193)	(8,899)
Non-interest Income
Mortgage banking activities, net	0	0
Investment fair value changes, net	(714)	(674)
Other income, net	126	147
Realized gains, net	0	0
Total non-interest income, net	(588)	(527)
General and administrative expenses	(15,148)	(17,504)
Loan acquisition costs	0	(4)
Other expenses	0	17
Benefit from (provision for) income taxes	0	0
Net Income	(24,929)	(26,917)
Non-cash amortization (expense) income, net	$ (2,033)	$ (1,895)