Accumulated Other Comprehensive Income (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2015	Dec. 31, 2014	Dec. 31, 2013
Accumulated Other Comprehensive Income Loss [Line Items]
Beginning balance accumulated other comprehensive income (loss)	$ 205,765	$ (8,052)	$ 262,854
Other comprehensive income (loss) before reclassification	(91,943)	140,325	(701,899)
Amount reclassified to MBS available for sale	(13,148)	(7,527)	23,032
Amounts reclassified to net gain/loss on the sale of MBS	(32,731)	(5,196)	283,012
Amounts reclassified for termination of all- in-one cash flow hedge accounting on dollar roll TBAs		5,083
Amounts reclassified for impairment loss on mortgage-backed securities			8,102
Amounts reclassified to interest expense	30,715	81,132	116,847
Net current period other comprehensive income (loss), unrealized gain/(loss) on derivative instruments	26,760	90,993	120,684
Other comprehensive income (loss)	(107,107)	213,817	(270,906)
Ending balance, total accumulated other comprehensive income (loss)	98,658	205,765	(8,052)
Derivative Instruments
Accumulated Other Comprehensive Income Loss [Line Items]
Beginning balance	(25,922)	(116,915)	(237,599)
Other comprehensive income (loss) before reclassification	8,275	11,789	(21,054)
Amount reclassified to MBS available for sale	(12,230)	(7,011)	24,891
Amounts reclassified for termination of all- in-one cash flow hedge accounting on dollar roll TBAs		5,083
Amounts reclassified to interest expense	30,715	81,132	116,847
Net current period other comprehensive income (loss), unrealized gain/(loss) on derivative instruments	26,760	90,993	120,684
Ending balance	838	(25,922)	(116,915)
Whole Mortgage Loans | Unrealized Gain (Loss) on Investments
Accumulated Other Comprehensive Income Loss [Line Items]
Beginning balance, accumulated other comprehensive income (loss), unrealized gain/(loss) on investments	232,240	109,490	499,343
Other comprehensive income (loss) before reclassification, unrealized gain/(loss) on derivative instruments	(101,141)	127,946	(680,967)
Amounts reclassified to net gain/loss on the sale of MBS	(32,731)	(5,196)	283,012
Amounts reclassified for impairment loss on mortgage-backed securities			8,102
Net current period other comprehensive income (loss), unrealized gain/(loss) on investments	(133,872)	122,750	(389,853)
Ending balance, accumulated other comprehensive income (loss), unrealized gain/(loss) on investments	98,368	232,240	109,490
Unsettled MBS | Unrealized Gain (Loss) on Investments
Accumulated Other Comprehensive Income Loss [Line Items]
Beginning balance, accumulated other comprehensive income (loss), unrealized gain/(loss) on investments	42		1,217
Other comprehensive income (loss) before reclassification, unrealized gain/(loss) on derivative instruments	875	558	642
Amount reclassified to MBS available for sale	(918)	(516)	(1,859)
Net current period other comprehensive income (loss), unrealized gain/(loss) on investments	(43)	42	(1,217)
Ending balance, accumulated other comprehensive income (loss), unrealized gain/(loss) on investments	(1)	42
Other Investments | Unrealized Gain (Loss) on Investments
Accumulated Other Comprehensive Income Loss [Line Items]
Beginning balance, accumulated other comprehensive income (loss), unrealized gain/(loss) on investments	(595)	(627)	(107)
Other comprehensive income (loss) before reclassification, unrealized gain/(loss) on derivative instruments	48	32	(520)
Net current period other comprehensive income (loss), unrealized gain/(loss) on investments	48	32	(520)
Ending balance, accumulated other comprehensive income (loss), unrealized gain/(loss) on investments	$ (547)	$ (595)	$ (627)