Agency Securities, Available for Sale - Unrealized Gain or Loss Position and Components of Carrying Value of Available for Sale Agency Securities (Details) - USD ($) $ in Thousands	3 Months Ended
	Mar. 31, 2016	Mar. 31, 2015	Dec. 31, 2015
Schedule of Available-for-sale Securities [Line Items]
Amortized Cost	$ 10,351,291		$ 12,528,858
Gross Unrealized Loss	(10,104)		(92,238)
Gross Unrealized Gain	132,823		24,936
Fair Value	$ 10,474,010		12,461,556
Portfolio Concentration Risk | Available-for-sale Securities
Schedule of Available-for-sale Securities [Line Items]
Percent of Total	100.00%	100.00%
Fannie Mae
Schedule of Available-for-sale Securities [Line Items]
Amortized Cost	$ 8,770,719		10,781,638
Gross Unrealized Loss	(3,916)		(74,115)
Gross Unrealized Gain	123,335		19,682
Fair Value	$ 8,890,138		10,727,205
Fannie Mae | Portfolio Concentration Risk | Available-for-sale Securities
Schedule of Available-for-sale Securities [Line Items]
Percent of Total	84.88%	86.08%
Fannie Mae | ARMs & Hybrids
Schedule of Available-for-sale Securities [Line Items]
Amortized Cost	$ 43,992		46,512
Gross Unrealized Loss	(103)		(210)
Gross Unrealized Gain	372		486
Fair Value	$ 44,261		46,788
Fannie Mae | ARMs & Hybrids | Portfolio Concentration Risk | Available-for-sale Securities
Schedule of Available-for-sale Securities [Line Items]
Percent of Total	0.42%	0.38%
Fannie Mae | Multi-Family MBS
Schedule of Available-for-sale Securities [Line Items]
Amortized Cost	$ 1,501,472		2,182,156
Gross Unrealized Loss	0		(30,879)
Gross Unrealized Gain	57,340		7,312
Fair Value	$ 1,558,812		2,158,589
Fannie Mae | Multi-Family MBS | Portfolio Concentration Risk | Available-for-sale Securities
Schedule of Available-for-sale Securities [Line Items]
Percent of Total	14.88%	17.32%
Fannie Mae | 10 Year Fixed
Schedule of Available-for-sale Securities [Line Items]
Amortized Cost	$ 171,556		91,752
Gross Unrealized Loss	(164)		(362)
Gross Unrealized Gain	1,539		605
Fair Value	$ 172,931		91,995
Fannie Mae | 10 Year Fixed | Portfolio Concentration Risk | Available-for-sale Securities
Schedule of Available-for-sale Securities [Line Items]
Percent of Total	1.65%	0.74%
Fannie Mae | 15 Year Fixed
Schedule of Available-for-sale Securities [Line Items]
Amortized Cost	$ 3,478,018		4,302,585
Gross Unrealized Loss	(134)		(10,462)
Gross Unrealized Gain	40,669		5,498
Fair Value	$ 3,518,553		4,297,621
Fannie Mae | 15 Year Fixed | Portfolio Concentration Risk | Available-for-sale Securities
Schedule of Available-for-sale Securities [Line Items]
Percent of Total	33.59%	34.49%
Fannie Mae | 20 Year Fixed
Schedule of Available-for-sale Securities [Line Items]
Amortized Cost	$ 2,340,415		2,692,310
Gross Unrealized Loss	(3,460)		(25,429)
Gross Unrealized Gain	13,266		5,289
Fair Value	$ 2,350,221		2,672,170
Fannie Mae | 20 Year Fixed | Portfolio Concentration Risk | Available-for-sale Securities
Schedule of Available-for-sale Securities [Line Items]
Percent of Total	22.44%	21.44%
Fannie Mae | 25 Year Fixed
Schedule of Available-for-sale Securities [Line Items]
Amortized Cost	$ 17,798		18,488
Gross Unrealized Loss	(10)		(128)
Gross Unrealized Gain	28		0
Fair Value	$ 17,816		18,360
Fannie Mae | 25 Year Fixed | Portfolio Concentration Risk | Available-for-sale Securities
Schedule of Available-for-sale Securities [Line Items]
Percent of Total	0.17%	0.15%
Fannie Mae | 30 Year Fixed
Schedule of Available-for-sale Securities [Line Items]
Amortized Cost	$ 1,217,468		1,447,835
Gross Unrealized Loss	(45)		(6,645)
Gross Unrealized Gain	10,121		492
Fair Value	$ 1,227,544		1,441,682
Fannie Mae | 30 Year Fixed | Portfolio Concentration Risk | Available-for-sale Securities
Schedule of Available-for-sale Securities [Line Items]
Percent of Total	11.73%	11.56%
Freddie Mac
Schedule of Available-for-sale Securities [Line Items]
Amortized Cost	$ 1,526,264		1,687,029
Gross Unrealized Loss	(5,459)		(17,513)
Gross Unrealized Gain	9,423		5,165
Fair Value	$ 1,530,228		1,674,681
Freddie Mac | Portfolio Concentration Risk | Available-for-sale Securities
Schedule of Available-for-sale Securities [Line Items]
Percent of Total	14.61%	13.44%
Freddie Mac | ARMs & Hybrids
Schedule of Available-for-sale Securities [Line Items]
Amortized Cost			12,738
Gross Unrealized Loss			(46)
Gross Unrealized Gain			197
Fair Value			12,889
Freddie Mac | ARMs & Hybrids | Portfolio Concentration Risk | Available-for-sale Securities
Schedule of Available-for-sale Securities [Line Items]
Percent of Total		0.10%
Freddie Mac | 10 Year Fixed
Schedule of Available-for-sale Securities [Line Items]
Amortized Cost	$ 81,795		37,657
Gross Unrealized Loss	(60)		(92)
Gross Unrealized Gain	1,538		652
Fair Value	$ 83,273		38,217
Freddie Mac | 10 Year Fixed | Portfolio Concentration Risk | Available-for-sale Securities
Schedule of Available-for-sale Securities [Line Items]
Percent of Total	0.80%	0.31%
Freddie Mac | 15 Year Fixed
Schedule of Available-for-sale Securities [Line Items]
Amortized Cost	$ 87,086		192,982
Gross Unrealized Loss	(1)		(995)
Gross Unrealized Gain	1,254		310
Fair Value	$ 88,339		192,297
Freddie Mac | 15 Year Fixed | Portfolio Concentration Risk | Available-for-sale Securities
Schedule of Available-for-sale Securities [Line Items]
Percent of Total	0.84%	1.54%
Freddie Mac | 20 Year Fixed
Schedule of Available-for-sale Securities [Line Items]
Amortized Cost	$ 1,357,383		1,443,652
Gross Unrealized Loss	(5,398)		(16,380)
Gross Unrealized Gain	6,631		4,006
Fair Value	$ 1,358,616		1,431,278
Freddie Mac | 20 Year Fixed | Portfolio Concentration Risk | Available-for-sale Securities
Schedule of Available-for-sale Securities [Line Items]
Percent of Total	12.97%	11.49%
Ginnie Mae
Schedule of Available-for-sale Securities [Line Items]
Amortized Cost	$ 54,308		60,191
Gross Unrealized Loss	(729)		(610)
Gross Unrealized Gain	65		89
Fair Value	$ 53,644		59,670
Ginnie Mae | Portfolio Concentration Risk | Available-for-sale Securities
Schedule of Available-for-sale Securities [Line Items]
Percent of Total	0.51%	0.48%
Ginnie Mae | ARMs & Hybrids
Schedule of Available-for-sale Securities [Line Items]
Amortized Cost	$ 54,000		59,877
Gross Unrealized Loss	(729)		(610)
Gross Unrealized Gain	43		69
Fair Value	$ 53,314		59,336
Ginnie Mae | ARMs & Hybrids | Portfolio Concentration Risk | Available-for-sale Securities
Schedule of Available-for-sale Securities [Line Items]
Percent of Total	0.51%	0.48%
Ginnie Mae | 15 Year Fixed
Schedule of Available-for-sale Securities [Line Items]
Amortized Cost	$ 308		314
Gross Unrealized Loss	0		0
Gross Unrealized Gain	22		20
Fair Value	$ 330		$ 334
Ginnie Mae | 15 Year Fixed | Portfolio Concentration Risk | Available-for-sale Securities
Schedule of Available-for-sale Securities [Line Items]
Percent of Total	0.00%	0.00%