Segment Information - Financial Information by Segment (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2023	Mar. 31, 2022
Segment Reporting Information [Line Items]
Interest income	$ 178,516	$ 189,400
Interest expense	(152,079)	(136,298)
Net Interest Income	26,437	53,102
Non-interest income (loss)
Mortgage banking activities, net	16,671	16,315
Investment fair value changes, net	(127)	(6,120)
Other income, net	4,556	5,983
Realized gains (losses), net	(2)	2,581
Total non-interest income, net	21,098	18,759
General and administrative expenses	(35,555)	(33,276)
Portfolio Management Costs	(3,510)	(1,578)
Loan acquisition costs	(1,289)	(4,465)
Other expenses	(3,684)	(4,085)
Benefit from (provision for) income taxes	1,123	2,458
Net Income	4,620	30,915
Non-cash amortization (expense) income, net	(8,895)	1,481
Operating Segments | Residential Mortgage Banking
Segment Reporting Information [Line Items]
Interest income	5,510	12,967
Interest expense	(6,866)	(6,936)
Net Interest Income	(1,356)	6,031
Non-interest income (loss)
Mortgage banking activities, net	3,365	7,935
Investment fair value changes, net	1,076	0
Other income, net	0	0
Realized gains (losses), net	0	0
Total non-interest income, net	4,441	7,935
General and administrative expenses	(4,806)	(6,101)
Portfolio Management Costs	0	0
Loan acquisition costs	(175)	(1,417)
Other expenses	0	0
Benefit from (provision for) income taxes	633	1,007
Net Income	(1,263)	7,455
Non-cash amortization (expense) income, net	(255)	104
Operating Segments | Business Purpose Mortgage Banking
Segment Reporting Information [Line Items]
Interest income	4,494	4,841
Interest expense	(4,038)	(2,568)
Net Interest Income	456	2,273
Non-interest income (loss)
Mortgage banking activities, net	13,306	8,380
Investment fair value changes, net	0	0
Other income, net	2,408	575
Realized gains (losses), net	0	0
Total non-interest income, net	15,714	8,955
General and administrative expenses	(13,678)	(10,472)
Portfolio Management Costs	0	0
Loan acquisition costs	(1,114)	(3,048)
Other expenses	(3,108)	(3,534)
Benefit from (provision for) income taxes	703	3,281
Net Income	(1,027)	(2,545)
Non-cash amortization (expense) income, net	(3,701)	(3,890)
Operating Segments | Investment Portfolio
Segment Reporting Information [Line Items]
Interest income	163,660	170,572
Interest expense	(123,452)	(116,581)
Net Interest Income	40,208	53,991
Non-interest income (loss)
Mortgage banking activities, net	0	0
Investment fair value changes, net	(1,014)	(5,406)
Other income, net	2,168	5,282
Realized gains (losses), net	(117)	2,581
Total non-interest income, net	1,037	2,457
General and administrative expenses	(1,409)	(1,555)
Portfolio Management Costs	(3,510)	(1,578)
Loan acquisition costs	0	0
Other expenses	(576)	(551)
Benefit from (provision for) income taxes	(213)	(1,830)
Net Income	35,537	50,934
Non-cash amortization (expense) income, net	(2,833)	7,300
Corporate/ Other
Segment Reporting Information [Line Items]
Interest income	4,852	1,020
Interest expense	(17,723)	(10,213)
Net Interest Income	(12,871)	(9,193)
Non-interest income (loss)
Mortgage banking activities, net	0	0
Investment fair value changes, net	(189)	(714)
Other income, net	(20)	126
Realized gains (losses), net	115	0
Total non-interest income, net	(94)	(588)
General and administrative expenses	(15,662)	(15,148)
Portfolio Management Costs	0	0
Loan acquisition costs	0	0
Other expenses	0	0
Benefit from (provision for) income taxes	0	0
Net Income	(28,627)	(24,929)
Non-cash amortization (expense) income, net	$ (2,106)	$ (2,033)