MBS and CRT Securities (Details 2)		3 Months Ended		9 Months Ended
	Dec. 31, 2014 USD ($)	Sep. 30, 2016 USD ($) Security	Sep. 30, 2015 USD ($)	Sep. 30, 2016 USD ($) Security	Sep. 30, 2015 USD ($)	Dec. 31, 2015 USD ($)
Fair Value
Less than 12 Months		$ 559,670,000		$ 559,670,000
12 Months or more		1,793,941,000		1,793,941,000
Total		2,353,611,000		2,353,611,000
Unrealized Losses
Less than 12 Months		1,629,000		1,629,000
12 Months or more		13,272,000		13,272,000
Total		$ 14,901,000		$ 14,901,000
Number of Securities
Less than 12 months (in security) | Security		92		92
12 months or more (in security) | Security		240		240
Gross Unrealized Losses		$ 14,901,000		$ 14,901,000		$ 71,929,000
Total OTTI losses		(1,255,000)	$ 0	(1,255,000)	$ (525,000)
OTTI recognized in/(reclassified from) OCI		770,000	0	770,000	(180,000)
Net Impairment Losses Recognized in Earnings		(485,000)	0	(485,000)	(705,000)
Roll-forward of the credit loss component of OTTI
Credit loss component of OTTI at beginning of period		36,820,000		36,820,000
Additions for credit related OTTI not previously recognized		314,000		314,000
Subsequent additional credit related OTTI recorded		171,000		171,000
Credit loss component of OTTI at end of period		37,305,000		37,305,000
Changes in the components of the purchase discount on Non-Agency MBS
Cumulative effect adjustment on adoption of revised accounting standard for repurchase agreement financing	$ 4,500,000	0	0	0	4,537,000
Proceeds from settlement				61,800,000
AOCI from AFS Securities:
Unrealized gain on AFS securities at beginning of period		625,697,000	759,151,000	585,250,000	813,515,000
Reclassification adjustment for MBS sales included in net income		(6,829,000)	(11,363,000)	(26,795,000)	(26,414,000)
Reclassification adjustment for OTTI included in net income		(485,000)	0	(485,000)	(705,000)
Change in AOCI from AFS securities		57,036,000	(55,402,000)	97,483,000	(109,766,000)
Balance at end of period	813,515,000	682,733,000	703,749,000	682,733,000	703,749,000
Total MBS
Fair Value
Less than 12 Months		559,670,000		559,670,000
12 Months or more		1,793,941,000		1,793,941,000
Total		2,353,611,000		2,353,611,000
Unrealized Losses
Less than 12 Months		1,629,000		1,629,000
12 Months or more		13,272,000		13,272,000
Total		$ 14,901,000		$ 14,901,000
Number of Securities
Less than 12 months (in security) | Security		92		92
12 months or more (in security) | Security		240		240
Gross Unrealized Losses		$ 14,901,000		$ 14,901,000		68,782,000
Agency MBS
Fair Value
Less than 12 Months		498,067,000		498,067,000
12 Months or more		1,178,396,000		1,178,396,000
Total		1,676,463,000		1,676,463,000
Unrealized Losses
Less than 12 Months		1,520,000		1,520,000
12 Months or more		8,513,000		8,513,000
Total		$ 10,033,000		$ 10,033,000
Number of Securities
Less than 12 months (in security) | Security		87		87
12 months or more (in security) | Security		202		202
Gross Unrealized Losses		$ 10,033,000		$ 10,033,000		40,425,000
AOCI from AFS Securities:
Unrealized (loss)/gain on AFS securities, net		(6,941,000)	(2,028,000)	17,857,000	(14,393,000)
MBS and CRT Securities Interest Income
Coupon interest		29,283,000	35,154,000	92,263,000	113,543,000
Effective yield adjustment		(10,326,000)	(11,536,000)	(27,717,000)	(32,513,000)
Interest income		18,957,000	23,618,000	64,546,000	81,030,000
Agency MBS | Fannie Mae
Fair Value
Less than 12 Months		362,752,000		362,752,000
12 Months or more		911,163,000		911,163,000
Total		1,273,915,000		1,273,915,000
Unrealized Losses
Less than 12 Months		1,335,000		1,335,000
12 Months or more		6,401,000		6,401,000
Total		$ 7,736,000		$ 7,736,000
Number of Securities
Less than 12 months (in security) | Security		75		75
12 months or more (in security) | Security		137		137
Gross Unrealized Losses		$ 7,736,000		$ 7,736,000		25,830,000
Agency MBS | Freddie Mac
Fair Value
Less than 12 Months		135,315,000		135,315,000
12 Months or more		267,233,000		267,233,000
Total		402,548,000		402,548,000
Unrealized Losses
Less than 12 Months		185,000		185,000
12 Months or more		2,112,000		2,112,000
Total		$ 2,297,000		$ 2,297,000
Number of Securities
Less than 12 months (in security) | Security		12		12
12 months or more (in security) | Security		65		65
Gross Unrealized Losses		$ 2,297,000		$ 2,297,000		14,595,000
Non-Agency MBS
Fair Value
Less than 12 Months		61,603,000		61,603,000
12 Months or more		615,545,000		615,545,000
Total		677,148,000		677,148,000
Unrealized Losses
Less than 12 Months		109,000		109,000
12 Months or more		4,759,000		4,759,000
Total		$ 4,868,000		$ 4,868,000
Number of Securities
Less than 12 months (in security) | Security		5		5
12 months or more (in security) | Security		38		38
Gross Unrealized Losses		$ 4,868,000		$ 4,868,000		28,357,000
Credit-related OTTI losses			0
AOCI from AFS Securities:
Unrealized (loss)/gain on AFS securities, net		71,291,000	(42,011,000)	106,906,000	(72,791,000)
Sales of MBS
Proceeds from sale of MBS categorized as available-for-sale		13,200,000	23,500,000	65,100,000	50,700,000
Gains on sales of MBS		7,100,000	11,200,000	26,100,000	25,200,000
Non-Agency MBS | Discount Designated as Credit Reserve and OTTI
Changes in the components of the purchase discount on Non-Agency MBS
Balance at beginning of period		(724,198,000)	(847,017,000)	(787,541,000)	(900,557,000)
Cumulative effect adjustment on adoption of revised accounting standard for repurchase agreement financing				0	(15,543,000)
Impact of RMBS Issuer settlement				0	0
Accretion of discount		0	0	0	0
Realized credit losses		15,629,000	21,527,000	49,408,000	62,377,000
Purchases		(15,124,000)	(455,000)	(25,999,000)	(1,200,000)
Sales		2,398,000	3,676,000	16,281,000	5,573,000
Net impairment losses recognized in earnings		(485,000)	0	(485,000)	(705,000)
Transfers/release of credit reserve		6,822,000	6,872,000	33,378,000	34,658,000
Balance at end of period	(900,557,000)	(714,958,000)	(815,397,000)	(714,958,000)	(815,397,000)
Non-Agency MBS | Accretable Discount
Changes in the components of the purchase discount on Non-Agency MBS
Balance at beginning of period		(325,548,000)	(362,946,000)	(312,182,000)	(399,564,000)
Cumulative effect adjustment on adoption of revised accounting standard for repurchase agreement financing				0	1,832,000
Impact of RMBS Issuer settlement				(52,881,000)	0
Accretion of discount		20,236,000	22,805,000	61,153,000	71,700,000
Realized credit losses		0	0	0	0
Purchases		9,830,000	113,000	13,210,000	(4,012,000)
Sales		6,523,000	11,193,000	28,297,000	28,995,000
Net impairment losses recognized in earnings		0	0	0	0
Transfers/release of credit reserve		(6,822,000)	(6,872,000)	(33,378,000)	(34,658,000)
Balance at end of period	$ (399,564,000)	(295,781,000)	(335,707,000)	(295,781,000)	(335,707,000)
Non-Agency MBS | Expected to Recover Par
Fair Value
Less than 12 Months		57,534,000		57,534,000
12 Months or more		525,183,000		525,183,000
Total		582,717,000		582,717,000
Unrealized Losses
Less than 12 Months		102,000		102,000
12 Months or more		1,804,000		1,804,000
Total		$ 1,906,000		$ 1,906,000
Number of Securities
Less than 12 months (in security) | Security		4		4
12 months or more (in security) | Security		20		20
Gross Unrealized Losses		$ 1,906,000		$ 1,906,000		20,143,000
Non-Agency MBS | Expected to Recover Less Than Par
Fair Value
Less than 12 Months		4,069,000		4,069,000
12 Months or more		90,362,000		90,362,000
Total		94,431,000		94,431,000
Unrealized Losses
Less than 12 Months		7,000		7,000
12 Months or more		2,955,000		2,955,000
Total		$ 2,962,000		$ 2,962,000
Number of Securities
Less than 12 months (in security) | Security		1		1
12 months or more (in security) | Security		18		18
Gross Unrealized Losses		$ 2,962,000		$ 2,962,000		8,214,000
Legacy Non-Agency MBS
MBS and CRT Securities Interest Income
Coupon interest		37,763,000	44,824,000	117,620,000	141,152,000
Effective yield adjustment		20,055,000	21,955,000	59,270,000	69,816,000
Interest income		57,818,000	66,779,000	176,890,000	210,968,000
3 Year Step-up Securities
MBS and CRT Securities Interest Income
Coupon interest		25,630,000	22,898,000	74,773,000	63,602,000
Effective yield adjustment		190,000	753,000	1,892,000	1,662,000
Interest income		25,820,000	23,651,000	76,665,000	65,264,000
CRT securities
Fair Value
Less than 12 Months		0		0
12 Months or more		0		0
Total		0		0
Unrealized Losses
Less than 12 Months		0		0
12 Months or more		0		0
Total		$ 0		$ 0
Number of Securities
Less than 12 months (in security) | Security		0		0
12 months or more (in security) | Security		0		0
Gross Unrealized Losses		$ 0		$ 0		3,147,000
MBS and CRT Securities Interest Income
Coupon interest		3,562,000	1,432,000	8,725,000	4,000,000
Effective yield adjustment		421,000	161,000	1,172,000	477,000
Interest income		$ 3,983,000	$ 1,593,000	$ 9,897,000	$ 4,477,000
CRT securities | Fair value option elected, debt securities
Number of Securities
Gross Unrealized Losses						$ 555,000