MBS and CRT Securities (Details) - USD ($) $ in Thousands	9 Months Ended	12 Months Ended
	Sep. 30, 2016	Dec. 31, 2015
Schedule of MBS and CRT Securities [Line Items]
Minimum term of fixed rate mortgages underlying MBS, years	15 years
Principal/ Current Face	$ 10,452,740	$ 11,703,896
Purchase Premiums	144,426	172,160
Accretable Purchase Discounts	(299,640)	(317,930)
Discount Designated as Credit Reserve and OTTI	(714,958)	(787,541)
Amortized Cost	9,584,332	10,771,616
Fair Value	10,277,990	11,356,643
Gross Unrealized Gains	708,559	656,956
Gross Unrealized Losses	(14,901)	(71,929)
Net Unrealized Gain	693,658	585,027
Principal payments receivable	1,800	1,000
Total MBS
Schedule of MBS and CRT Securities [Line Items]
Principal/ Current Face	10,121,012	11,511,896
Purchase Premiums	144,426	172,160
Accretable Purchase Discounts	(295,834)	(312,241)
Discount Designated as Credit Reserve and OTTI	(714,958)	(787,541)
Amortized Cost	9,256,410	10,585,305
Fair Value	9,930,350	11,173,061
Gross Unrealized Gains	688,841	656,538
Gross Unrealized Losses	(14,901)	(68,782)
Net Unrealized Gain	673,940	587,756
Agency MBS
Schedule of MBS and CRT Securities [Line Items]
Principal/ Current Face	3,828,822	4,550,403
Purchase Premiums	144,363	172,087
Accretable Purchase Discounts	(53)	(59)
Discount Designated as Credit Reserve and OTTI	0	0
Amortized Cost	3,974,896	4,723,462
Fair Value	4,021,535	4,752,244
Gross Unrealized Gains	56,672	69,207
Gross Unrealized Losses	(10,033)	(40,425)
Net Unrealized Gain	46,639	28,782
Agency MBS | Fannie Mae
Schedule of MBS and CRT Securities [Line Items]
Principal/ Current Face	3,084,420	3,690,020
Purchase Premiums	115,925	139,243
Accretable Purchase Discounts	(53)	(59)
Discount Designated as Credit Reserve and OTTI	0	0
Amortized Cost	3,200,292	3,829,204
Fair Value	3,242,427	3,865,485
Gross Unrealized Gains	49,871	62,111
Gross Unrealized Losses	(7,736)	(25,830)
Net Unrealized Gain	42,135	36,281
Agency MBS | Freddie Mac
Schedule of MBS and CRT Securities [Line Items]
Principal/ Current Face	736,602	851,087
Purchase Premiums	28,298	32,680
Accretable Purchase Discounts	0	0
Discount Designated as Credit Reserve and OTTI	0	0
Amortized Cost	766,664	884,798
Fair Value	771,016	877,109
Gross Unrealized Gains	6,649	6,906
Gross Unrealized Losses	(2,297)	(14,595)
Net Unrealized Gain	4,352	(7,689)
Agency MBS | Ginnie Mae
Schedule of MBS and CRT Securities [Line Items]
Principal/ Current Face	7,800	9,296
Purchase Premiums	140	164
Accretable Purchase Discounts	0	0
Discount Designated as Credit Reserve and OTTI	0	0
Amortized Cost	7,940	9,460
Fair Value	8,092	9,650
Gross Unrealized Gains	152	190
Gross Unrealized Losses	0	0
Net Unrealized Gain	152	190
Non-Agency MBS
Schedule of MBS and CRT Securities [Line Items]
Principal/ Current Face	6,292,190	6,961,493
Purchase Premiums	63	73
Accretable Purchase Discounts	(295,781)	(312,182)
Discount Designated as Credit Reserve and OTTI	(714,958)	(787,541)
Amortized Cost	5,281,514	5,861,843
Fair Value	5,908,815	6,420,817
Gross Unrealized Gains	632,169	587,331
Gross Unrealized Losses	(4,868)	(28,357)
Net Unrealized Gain	627,301	558,974
Credit Reserve	694,000	766,000
OTTI	20,900	21,500
Non-Agency MBS | Expected to Recover Par
Schedule of MBS and CRT Securities [Line Items]
Principal/ Current Face	2,799,010	2,906,878
Purchase Premiums	63	73
Accretable Purchase Discounts	(37,438)	(31,576)
Discount Designated as Credit Reserve and OTTI	0	0
Amortized Cost	2,761,635	2,875,375
Fair Value	2,801,189	2,878,532
Gross Unrealized Gains	41,460	23,300
Gross Unrealized Losses	(1,906)	(20,143)
Net Unrealized Gain	39,554	3,157
Non-Agency MBS | Expected to Recover Less Than Par
Schedule of MBS and CRT Securities [Line Items]
Principal/ Current Face	3,493,180	4,054,615
Purchase Premiums	0	0
Accretable Purchase Discounts	(258,343)	(280,606)
Discount Designated as Credit Reserve and OTTI	(714,958)	(787,541)
Amortized Cost	2,519,879	2,986,468
Fair Value	3,107,626	3,542,285
Gross Unrealized Gains	590,709	564,031
Gross Unrealized Losses	(2,962)	(8,214)
Net Unrealized Gain	$ 587,747	$ 555,817
Percentage of current face amount of Non-Agency MBS to be recovered	89.00%	89.00%
CRT securities
Schedule of MBS and CRT Securities [Line Items]
Principal/ Current Face	$ 331,728	$ 192,000
Purchase Premiums	0	0
Accretable Purchase Discounts	(3,806)	(5,689)
Discount Designated as Credit Reserve and OTTI	0	0
Amortized Cost	327,922	186,311
Fair Value	347,640	183,582
Gross Unrealized Gains	19,718	418
Gross Unrealized Losses	0	(3,147)
Net Unrealized Gain	19,718	(2,729)
CRT securities | Fair value option elected, debt securities
Schedule of MBS and CRT Securities [Line Items]
Fair Value	214,600	62,200
Gross Unrealized Gains	$ 10,900	332
Gross Unrealized Losses		(555)
Net Unrealized Gain		(223)
3 Year Step-up Securities
Schedule of MBS and CRT Securities [Line Items]
Debt instrument, basis spread on variable rate	3.00%
Debt instrument, coupon step-up period	36 months
Principal/ Current Face	$ 2,500,000	2,600,000
Amortized Cost	2,500,000	2,600,000
Fair Value	$ 2,500,000	$ 2,600,000