Segment Information - Financial Information by Segment (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2019	Sep. 30, 2018	Sep. 30, 2019	Sep. 30, 2018
Segment Reporting Information [Line Items]
Interest income	$ 150,117	$ 99,397	$ 429,700	$ 258,992
Interest expense	(116,604)	(64,351)	(332,100)	(154,078)
Net Interest Income	33,513	35,046	97,600	104,914
Non-interest Income
Mortgage banking activities, net	9,515	11,224	40,984	48,396
Investment fair value changes, net	11,444	10,332	34,741	12,830
Other income (expense), net	1,825	3,453	7,819	8,893
Realized gains, net	4,714	7,275	18,227	21,352
Total non-interest income, net	27,498	32,284	101,771	91,471
Direct operating expenses	(26,815)	(21,490)	(76,229)	(63,529)
(Provision for) benefit from income taxes	114	(4,919)	(3,102)	(12,343)
Net Income	34,310	40,921	120,040	120,513
Non-cash amortization income (expense), net	(720)	2,789	(892)	10,170
Operating Segments | Investment Portfolio
Segment Reporting Information [Line Items]
Interest income	132,894	79,556	380,394	202,882
Interest expense	(94,519)	(40,852)	(266,318)	(87,719)
Net Interest Income	38,375	38,704	114,076	115,163
Non-interest Income
Mortgage banking activities, net	0	0	0	0
Investment fair value changes, net	11,896	10,566	35,749	13,756
Other income (expense), net	2,313	3,334	6,408	8,774
Realized gains, net	4,714	7,275	18,227	21,352
Total non-interest income, net	18,923	21,175	60,384	43,882
Direct operating expenses	(2,191)	(2,659)	(7,110)	(6,524)
(Provision for) benefit from income taxes	(89)	(2,840)	(1,327)	(4,858)
Net Income	55,018	54,380	166,023	147,663
Non-cash amortization income (expense), net	2,456	4,019	7,446	13,290
Operating Segments | Mortgage Banking
Segment Reporting Information [Line Items]
Interest income	12,491	14,427	34,220	40,408
Interest expense	(6,657)	(7,537)	(18,816)	(21,303)
Net Interest Income	5,834	6,890	15,404	19,105
Non-interest Income
Mortgage banking activities, net	9,515	11,224	40,984	48,396
Investment fair value changes, net	0	0	0	0
Other income (expense), net	(252)	0	(575)	0
Realized gains, net	0	0	0	0
Total non-interest income, net	9,263	11,224	40,409	48,396
Direct operating expenses	(11,907)	(6,570)	(31,582)	(20,941)
(Provision for) benefit from income taxes	203	(2,079)	(1,775)	(7,485)
Net Income	3,393	9,465	22,456	39,075
Non-cash amortization income (expense), net	(2,028)	(54)	(4,765)	(99)
Corporate/ Other
Segment Reporting Information [Line Items]
Interest income	4,732	5,414	15,086	15,702
Interest expense	(15,428)	(15,962)	(46,966)	(45,056)
Net Interest Income	(10,696)	(10,548)	(31,880)	(29,354)
Non-interest Income
Mortgage banking activities, net	0	0	0	0
Investment fair value changes, net	(452)	(234)	(1,008)	(926)
Other income (expense), net	(236)	119	1,986	119
Realized gains, net	0	0	0	0
Total non-interest income, net	(688)	(115)	978	(807)
Direct operating expenses	(12,717)	(12,261)	(37,537)	(36,064)
(Provision for) benefit from income taxes	0	0	0	0
Net Income	(24,101)	(22,924)	(68,439)	(66,225)
Non-cash amortization income (expense), net	$ (1,148)	$ (1,176)	$ (3,573)	$ (3,021)