Accumulated Other Comprehensive Income (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2015	Sep. 30, 2014	Sep. 30, 2015	Sep. 30, 2014
Accumulated Other Comprehensive Income Loss [Line Items]
Beginning balance accumulated other comprehensive income (loss)	$ 226,809	$ 164,223	$ 205,765	$ (8,052)
Other comprehensive income (loss) before reclassification	1,420	(28,691)	25,695	94,944
Amount reclassified to MBS available for sale	(1,150)	(4,337)	(10,954)	(5,535)
Amounts reclassified to net gain/loss on the sale of MBS	(1,216)	(237)	(17,360)	(3,089)
Amounts reclassified to interest expense	5,433	19,806	28,150	67,413
Net current period other comprehensive income (loss), unrealized gain/(loss) on investments	(912)		(912)
Amounts reclassified for termination of all- in-one cash flow hedge accounting on dollar roll TBAs				5,083
Net current period other comprehensive income (loss), unrealized gain/(loss) on derivative instruments	6,389	17,923	26,139	75,026
Other comprehensive income (loss)	4,487	(13,459)	25,531	158,816
Ending balance, total accumulated other comprehensive income (loss)	231,296	150,764	231,296	150,764
Derivative Instruments
Accumulated Other Comprehensive Income Loss [Line Items]
Beginning balance	(6,172)	(59,812)	(25,922)	(116,915)
Other comprehensive income (loss) before reclassification	2,109	1,872	8,110	7,669
Amounts reclassified to interest expense	5,433	19,806	28,150	67,413
Amount reclassified to MBS available for sale	(1,153)	(3,755)	(10,121)	(5,139)
Amounts reclassified for termination of all- in-one cash flow hedge accounting on dollar roll TBAs				5,083
Net current period other comprehensive income (loss), unrealized gain/(loss) on derivative instruments	6,389	17,923	26,139	75,026
Ending balance	217	(41,889)	217	(41,889)
Whole Mortgage Loans | Unrealized Gain (Loss) on Investments
Accumulated Other Comprehensive Income Loss [Line Items]
Beginning balance, accumulated other comprehensive income (loss), unrealized gain/(loss) on investments	233,289	223,948	232,240	109,490
Other comprehensive income (loss) before reclassification, unrealized gain/(loss) on derivative instruments	(753)	(30,572)	16,440	86,738
Amounts reclassified to net gain/loss on the sale of MBS	(1,216)	(237)	(17,360)	(3,089)
Net current period other comprehensive income (loss), unrealized gain/(loss) on investments	(1,969)	(30,809)	(920)	83,649
Ending balance, accumulated other comprehensive income (loss), unrealized gain/(loss) on investments	231,320	193,139	231,320	193,139
Unsettled MBS | Unrealized Gain (Loss) on Investments
Accumulated Other Comprehensive Income Loss [Line Items]
Beginning balance, accumulated other comprehensive income (loss), unrealized gain/(loss) on investments	(3)	582	42
Other comprehensive income (loss) before reclassification, unrealized gain/(loss) on derivative instruments	85	120	876	516
Net current period other comprehensive income (loss), unrealized gain/(loss) on investments	88	(462)	43	120
Ending balance, accumulated other comprehensive income (loss), unrealized gain/(loss) on investments	85	120	85	120
Amount reclassified to MBS available for sale	3	(582)	(833)	(396)
Other Investments | Unrealized Gain (Loss) on Investments
Accumulated Other Comprehensive Income Loss [Line Items]
Beginning balance, accumulated other comprehensive income (loss), unrealized gain/(loss) on investments	(305)	(495)	(595)	(627)
Other comprehensive income (loss) before reclassification, unrealized gain/(loss) on derivative instruments	(21)	(111)	269	21
Net current period other comprehensive income (loss), unrealized gain/(loss) on investments	(21)	(111)	269	21
Ending balance, accumulated other comprehensive income (loss), unrealized gain/(loss) on investments	$ (326)	$ (606)	$ (326)	$ (606)