Accumulated Other Comprehensive Income (Details) (USD $) In Thousands, unless otherwise specified	3 Months Ended
	Mar. 31, 2014	Mar. 31, 2013
Accumulated Other Comprehensive Income Loss [Line Items]
Beginning balance accumulated other comprehensive income (loss)	$ (8,052)	$ 262,854
Other comprehensive income (loss) before reclassifications	43,616	(14,566)
Amounts reclassified to net gain on the sale of mortgage-backed securities	658	(6,669)
Amounts reclassified for termination of all-in-one cash flow hedge accounting on dollar roll TBAs	5,083
Amounts reclassified to net gain on the sale of mortgage-backed securities	(7,436)	(2,500)
Amounts reclassified to interest expense	24,684	29,929
Net current period other comprehensive income (loss)	32,467	25,046
Other comprehensive income (loss)	66,605	6,194
Ending balance accumulated other comprehensive income (loss)	58,553	269,048
Derivative Instruments
Accumulated Other Comprehensive Income Loss [Line Items]
Beginning balance	(116,915)	(237,599)
Amounts reclassified for termination of all-in-one cash flow hedge accounting on dollar roll TBAs	5,083
Other comprehensive income (loss) before reclassification	2,042	569
Amount reclassified to mortgage-backed securities available for sale	658	(5,452)
Amounts reclassified to net gain on the sale of mortgage-backed securities
Amounts reclassified to interest expense	24,684	29,929
Net current period other comprehensive income (loss)	32,467	25,046
Ending balance	(84,448)	(212,553)
Other Investments
Accumulated Other Comprehensive Income Loss [Line Items]
Beginning balance accumulated other comprehensive income (loss)	(628)	(107)
Other comprehensive income (loss) before reclassification	59	(107)
Amount reclassified to mortgage-backed securities available for sale
Amounts reclassified to net gain on the sale of mortgage-backed securities
Amounts reclassified for termination of all-in-one cash flow hedge accounting on dollar roll TBAs
Net current period other comprehensive income (loss)	59	(107)
Ending balance accumulated other comprehensive income (loss)	(569)	(214)
Available-for-sale Securities
Accumulated Other Comprehensive Income Loss [Line Items]
Beginning balance accumulated other comprehensive income (loss)	109,491	499,343
Other comprehensive income (loss) before reclassification	41,701	(15,520)
Amount reclassified to mortgage-backed securities available for sale
Amounts reclassified to net gain on the sale of mortgage-backed securities	(7,436)	(2,500)
Amounts reclassified for termination of all-in-one cash flow hedge accounting on dollar roll TBAs
Net current period other comprehensive income (loss)	34,265	(18,020)
Ending balance accumulated other comprehensive income (loss)	143,756	481,323
Unsettled Agency Contracts
Accumulated Other Comprehensive Income Loss [Line Items]
Beginning balance accumulated other comprehensive income (loss)		1,217
Other comprehensive income (loss) before reclassification	(186)	492
Amount reclassified to mortgage-backed securities available for sale		(1,217)
Amounts reclassified to net gain on the sale of mortgage-backed securities
Amounts reclassified for termination of all-in-one cash flow hedge accounting on dollar roll TBAs
Net current period other comprehensive income (loss)	(186)	(725)
Ending balance accumulated other comprehensive income (loss)	$ (186)	$ 492