Segment Information - Financial Information by Segment (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2017	Sep. 30, 2017	Jun. 30, 2017	Mar. 31, 2017	Dec. 31, 2016	Sep. 30, 2016	Jun. 30, 2016	Mar. 31, 2016	Dec. 31, 2017	Dec. 31, 2016	Dec. 31, 2015
Segment Reporting Information [Line Items]
Interest income	$ 71,468	$ 62,737	$ 59,224	$ 54,628	$ 56,334	$ 60,906	$ 66,787	$ 62,328	$ 248,057	$ 246,355	$ 259,432
Interest expense	(36,108)	(27,443)	(24,234)	(21,031)	(20,537)	(21,597)	(22,444)	(23,950)	(108,816)	(88,528)	(95,883)
Net Interest Income	35,360	35,294	34,990	33,597	35,797	39,309	44,343	38,378	139,241	157,827	163,549
Reversal of provision for loan losses					0	859	6,532	(289)	0	7,102	355
Non-interest income
Mortgage banking activities, net									53,908	38,691	10,972
MSR income (loss), net									7,860	14,353	(3,922)
Investment fair value changes, net									10,374	(28,574)	(21,357)
Other income									4,576	6,338	3,192
Realized gains, net									13,355	28,009	36,369
Total non-interest income, net	10,951	26,070	25,297	27,755	9,763	33,712	10,888	4,454	90,073	58,817	25,254
Direct operating expenses	(20,367)	(19,922)	(18,641)	(18,226)	(17,824)	(20,355)	(20,155)	(30,452)	(77,156)	(88,786)	(97,416)
Benefit from income taxes									(11,752)	(3,708)	10,346
Net Income	$ 30,933	$ 36,180	$ 36,324	$ 36,969	$ 25,355	$ 52,553	$ 41,281	$ 12,063	140,406	131,252	102,088
Non-cash amortization income (expense), net									17,462	25,704	32,403
Hedging allocations											0
Operating Segments | Investment Portfolio
Segment Reporting Information [Line Items]
Interest income									188,760	192,200	177,595
Interest expense									(36,690)	(22,997)	(22,684)
Net Interest Income									152,070	169,203	154,911
Reversal of provision for loan losses										7,102	355
Non-interest income
Mortgage banking activities, net									0	0	0
MSR income (loss), net									7,860	14,353	(3,922)
Investment fair value changes, net									18,414	(24,367)	(20,089)
Other income									4,576	6,338	3,192
Realized gains, net									14,107	27,717	36,369
Total non-interest income, net									44,957	24,041	15,550
Direct operating expenses									(6,028)	(10,421)	(7,179)
Benefit from income taxes									(5,328)	(1,848)	846
Net Income									185,671	188,077	164,483
Non-cash amortization income (expense), net									20,974	29,806	36,583
Hedging allocations											(1,070)
Operating Segments | Residential Mortgage Banking
Segment Reporting Information [Line Items]
Interest income									39,309	33,661	52,260
Interest expense									(17,369)	(14,191)	(17,207)
Net Interest Income									21,940	19,470	35,053
Reversal of provision for loan losses										0	0
Non-interest income
Mortgage banking activities, net									53,908	40,753	8,268
MSR income (loss), net									0	0	0
Investment fair value changes, net									0	0	0
Other income									0	0	0
Realized gains, net									0	0	0
Total non-interest income, net									53,908	40,753	8,268
Direct operating expenses									(25,113)	(23,252)	(43,182)
Benefit from income taxes									(6,424)	(1,860)	4,169
Net Income									44,311	35,111	4,308
Non-cash amortization income (expense), net									(102)	(130)	(186)
Hedging allocations											1,120
Corporate/ Other
Segment Reporting Information [Line Items]
Interest income									19,988	20,494	29,577
Interest expense									(54,757)	(51,340)	(55,992)
Net Interest Income									(34,769)	(30,846)	(26,415)
Reversal of provision for loan losses										0	0
Non-interest income
Mortgage banking activities, net									0	(2,062)	2,704
MSR income (loss), net									0	0	0
Investment fair value changes, net									(8,040)	(4,207)	(1,268)
Other income									0	0	0
Realized gains, net									(752)	292	0
Total non-interest income, net									(8,792)	(5,977)	1,436
Direct operating expenses									(46,015)	(55,113)	(47,055)
Benefit from income taxes									0	0	5,331
Net Income									(89,576)	(91,936)	(66,703)
Non-cash amortization income (expense), net									$ (3,410)	$ (3,972)	(3,994)
Hedging allocations											$ (50)