Accumulated Other Comprehensive Income (Details) (USD $) In Thousands, unless otherwise specified	3 Months Ended		6 Months Ended
	Jun. 30, 2014	Jun. 30, 2013	Jun. 30, 2014	Jun. 30, 2013
Accumulated Other Comprehensive Income Loss [Line Items]
Amounts reclassified to net gain on the sale of mortgage-backed securities	$ (1,856)	$ (6,968)	$ (1,198)	$ (13,637)
Amounts reclassified for termination of all- in-one cash flow hedge accounting on dollar roll TBAs			5,083
Amounts reclassified to interest expense	22,923	30,586	47,607	60,515
Amounts reclassified to net loss on the sale of mortgage-backed securities	4,584	(8,802)	(2,852)	(11,302)
Net current period other comprehensive income (loss)	24,636	37,679	57,103	62,725
Beginning balance accumulated other comprehensive income (loss)	58,553	269,048	(8,052)	262,854
Other comprehensive income (loss) before reclassification	80,019	(604,697)	123,635	(619,263)
Net current period other comprehensive income (loss)	105,670	(589,881)	172,275	(583,687)
Ending balance accumulated other comprehensive income (loss)	164,223	(320,833)	164,223	(320,833)
Derivative Instruments
Accumulated Other Comprehensive Income Loss [Line Items]
Amounts reclassified for termination of all- in-one cash flow hedge accounting on dollar roll TBAs			5,083
Amounts reclassified to interest expense	22,923	30,586	47,607	60,515
Beginning balance	(84,448)	(212,553)	(116,915)	(237,599)
Other comprehensive income (loss) before reclassification	3,755	13,569	5,797	14,138
Amount reclassified to mortgage-backed securities available for sale	(2,042)	(6,476)	(1,384)	(11,928)
Net current period other comprehensive income (loss)	24,636	37,679	57,103	62,725
Ending balance	(59,812)	(174,874)	(59,812)	(174,874)
Other Investments
Accumulated Other Comprehensive Income Loss [Line Items]
Beginning balance accumulated other comprehensive income (loss)	(569)	(214)	(627)	(107)
Other comprehensive income (loss) before reclassification	74	(291)	132	(398)
Amounts reclassified for termination of all- in-one cash flow hedge accounting on dollar roll TBAs			0
Net current period other comprehensive income (loss)	74	(291)	132	(398)
Ending balance accumulated other comprehensive income (loss)	(495)	(505)	(495)	(505)
Available-for-sale Securities
Accumulated Other Comprehensive Income Loss [Line Items]
Beginning balance accumulated other comprehensive income (loss)	143,756	481,323	109,490	499,343
Other comprehensive income (loss) before reclassification	75,608	(615,199)	117,310	(630,719)
Amounts reclassified to net gain on the sale of mortgage-backed securities	4,584	(8,802)	(2,852)	(11,302)
Amounts reclassified for termination of all- in-one cash flow hedge accounting on dollar roll TBAs			0
Net current period other comprehensive income (loss)	80,192	(624,001)	114,458	(642,021)
Ending balance accumulated other comprehensive income (loss)	223,948	(142,678)	223,948	(142,678)
Unsettled Agency Contracts
Accumulated Other Comprehensive Income Loss [Line Items]
Beginning balance accumulated other comprehensive income (loss)	(186)	492		1,217
Other comprehensive income (loss) before reclassification	582	(2,776)	396	(2,284)
Amount reclassified to mortgage-backed securities available for sale	186	(492)	186	(1,709)
Amounts reclassified for termination of all- in-one cash flow hedge accounting on dollar roll TBAs			0
Net current period other comprehensive income (loss)	768	(3,268)	582	(3,993)
Ending balance accumulated other comprehensive income (loss)	$ 582	$ (2,776)	$ 582	$ (2,776)