MBS and CRT Securities (Details 2)		3 Months Ended				12 Months Ended
		Dec. 31, 2015 USD ($) Security	Sep. 30, 2015 USD ($)	Jun. 30, 2015 USD ($)	Mar. 31, 2015 USD ($)	Dec. 31, 2015 USD ($) Security	Dec. 31, 2014 USD ($)		Dec. 31, 2013 USD ($)
Fair Value
Less than 12 months		$ 3,717,037,000				$ 3,717,037,000
12 months or more		1,410,241,000				1,410,241,000
Total		5,127,278,000				5,127,278,000
Unrealized Losses
Less than 12 months		36,748,000				36,748,000
12 months or more		35,181,000				35,181,000
Total		$ 71,929,000				$ 71,929,000
Number of Securities
Less than 12 Months | Security		290				290
12 Months or more | Security		197				197
OTTI charges recorded
Total OTTI losses						$ (525,000)	$ 0		$ 0
OTTI reclassified from OCI						(180,000)	0		0
Net Impairment Losses Recognized in Earnings		$ 0	$ 0	$ (298,000)	$ (407,000)	(705,000)	0		0
Roll-forward of the credit loss component of OTTI
Credit loss component of OTTI at beginning of period					36,115,000	36,115,000	36,115,000		36,115,000
Additions for credit related OTTI not previously recognized						461,000	0		0
Subsequent additional credit related OTTI recorded						244,000	0		0
Credit loss component of OTTI at end of period		36,820,000				36,820,000	36,115,000		36,115,000
Changes in the components of the purchase discount on Non-Agency MBS
Cumulative effect adjustment on adoption of revised accounting standard for repurchase agreement financing						4,537,000	0		0
Accumulated other comprehensive income from MBS and CRT securities:
Unrealized gain on AFS securities at beginning of period					813,515,000	813,515,000	752,912,000		824,808,000
Cumulative effect adjustment on adoption of revised accounting standard for repurchase agreement financing						4,537,000	0		0
Reclassification adjustment for MBS sales included in net income						(37,207,000)	(34,948,000)		(19,833,000)
Reclassification adjustment for OTTI included in net income						(705,000)	0		0
Change in AOCI from AFS securities						(228,265,000)	60,603,000		(71,896,000)
Balance at end of period		585,250,000				585,250,000	813,515,000		752,912,000
Total MBS
Fair Value
Less than 12 months		3,579,452,000				3,579,452,000
12 months or more		1,405,716,000				1,405,716,000
Total		4,985,168,000				4,985,168,000
Unrealized Losses
Less than 12 months		34,076,000				34,076,000
12 months or more		34,706,000				34,706,000
Total		$ 68,782,000				$ 68,782,000
Number of Securities
Less than 12 Months | Security		257				257
12 Months or more | Security		196				196
Agency MBS
Fair Value
Less than 12 months		$ 1,155,370,000				$ 1,155,370,000
12 months or more		1,129,051,000				1,129,051,000
Total		2,284,421,000				2,284,421,000
Unrealized Losses
Less than 12 months		13,011,000				13,011,000
12 months or more		27,414,000				27,414,000
Total		$ 40,425,000				$ 40,425,000
Number of Securities
Less than 12 Months | Security		163				163
12 Months or more | Security		173				173
Accumulated other comprehensive income from MBS and CRT securities:
Unrealized (loss)/gain on MBS and CRT securities, net						$ (51,332,000)	65,739,000		(186,568,000)
MBS and CRT securities Interest Income
Coupon interest						147,066,000	189,355,000		213,995,000
Effective yield adjustment	[1]					(41,231,000)	(46,812,000)		(57,949,000)
Interest income						105,835,000	142,543,000		156,046,000
Agency MBS | Fannie Mae
Fair Value
Less than 12 months		$ 856,602,000				856,602,000
12 months or more		813,485,000				813,485,000
Total		1,670,087,000				1,670,087,000
Unrealized Losses
Less than 12 months		7,548,000				7,548,000
12 months or more		18,282,000				18,282,000
Total		$ 25,830,000				$ 25,830,000
Number of Securities
Less than 12 Months | Security		121				121
12 Months or more | Security		109				109
Agency MBS | Freddie Mac
Fair Value
Less than 12 months		$ 298,768,000				$ 298,768,000
12 months or more		315,566,000				315,566,000
Total		614,334,000				614,334,000
Unrealized Losses
Less than 12 months		5,463,000				5,463,000
12 months or more		9,132,000				9,132,000
Total		$ 14,595,000				$ 14,595,000
Number of Securities
Less than 12 Months | Security		42				42
12 Months or more | Security		64				64
Non-Agency MBS
Fair Value
Less than 12 months		$ 2,424,082,000				$ 2,424,082,000
12 months or more		276,665,000				276,665,000
Total		2,700,747,000				2,700,747,000
Unrealized Losses
Less than 12 months		21,065,000				21,065,000
12 months or more		7,292,000				7,292,000
Total		$ 28,357,000				$ 28,357,000
Number of Securities
Less than 12 Months | Security		94				94
12 Months or more | Security		23				23
Credit-related OTTI losses						$ 705,000	0		0
Changes in the components of the purchase discount on Non-Agency MBS
Purchase discount pertaining to linked transactions designated as accretable purchase discount reallocated to credit reserve							218,000
Accumulated other comprehensive income from MBS and CRT securities:
Unrealized (loss)/gain on MBS and CRT securities, net						(143,558,000)	29,812,000		134,505,000
Sales of MBS
Proceeds from Sale of Mortgage Backed Securities (MBS) categorized as Available-for-sale						70,700,000	123,900,000		152,600,000
Gain (Loss) on Sales of Mortgage Backed Securities (MBS)						34,900,000	37,500,000		25,800,000
Non-Agency MBS | Discount Designated as Credit Reserve and OTTI
Changes in the components of the purchase discount on Non-Agency MBS
Balance at beginning of period	[2]				(900,557,000)	(900,557,000)	(1,043,037,000)
Cumulative effect adjustment on adoption of revised accounting standard for repurchase agreement financing						(15,543,000)	0	[2]
Accretion of discount						0	0	[2]
Realized credit losses						80,821,000	89,481,000	[2]
Purchases						(1,200,000)	(80,256,000)	[2]
Sales						8,525,000	44,692,000	[2]
Unlinking of Linked Transactions						0	(6,414,000)	[2]
Transfers/release of credit reserve						41,118,000	94,977,000	[2]
Balance at end of period		$ (787,541,000)				(787,541,000)	(900,557,000)	[2]	(1,043,037,000)	[2]
Accumulated other comprehensive income from MBS and CRT securities:
Cumulative effect adjustment on adoption of revised accounting standard for repurchase agreement financing						(15,543,000)	0	[2]
MBS and CRT securities Interest Income
Available For Sale Securities Changes in Purchase Discount Due to Net Impairment Losses Recognized in Earnings						(705,000)	0	[2]
Non-Agency MBS | Accretable Discount
Changes in the components of the purchase discount on Non-Agency MBS
Balance at beginning of period	[2],[3]				$ (399,564,000)	(399,564,000)	(460,039,000)
Cumulative effect adjustment on adoption of revised accounting standard for repurchase agreement financing	[3]					1,832,000	0	[2]
Accretion of discount	[3]					93,173,000	103,653,000	[2]
Realized credit losses	[3]					0	0	[2]
Purchases	[3]					(4,925,000)	30,003,000	[2]
Sales	[3]					38,420,000	20,360,000	[2]
Unlinking of Linked Transactions	[3]					0	1,436,000	[2]
Transfers/release of credit reserve	[3]					(41,118,000)	(94,977,000)	[2]
Balance at end of period	[3]	(312,182,000)				(312,182,000)	(399,564,000)	[2]	(460,039,000)	[2]
Accumulated other comprehensive income from MBS and CRT securities:
Cumulative effect adjustment on adoption of revised accounting standard for repurchase agreement financing	[3]					1,832,000	0	[2]
MBS and CRT securities Interest Income
Available For Sale Securities Changes in Purchase Discount Due to Net Impairment Losses Recognized in Earnings	[3]					0	0	[2]
Non-Agency MBS | Expected to Recover Par
Fair Value
Less than 12 months	[4]	2,239,418,000				2,239,418,000
12 months or more	[4]	212,584,000				212,584,000
Total	[4]	2,452,002,000				2,452,002,000
Unrealized Losses
Less than 12 months	[4]	16,717,000				16,717,000
12 months or more	[4]	3,426,000				3,426,000
Total	[4]	$ 20,143,000				$ 20,143,000
Number of Securities
Less than 12 Months | Security	[4]	59				59
12 Months or more | Security	[4]	12				12
Non-Agency MBS | Expected to Recover Less Than Par
Fair Value
Less than 12 months	[4]	$ 184,664,000				$ 184,664,000
12 months or more	[4]	64,081,000				64,081,000
Total	[4]	248,745,000				248,745,000
Unrealized Losses
Less than 12 months	[4]	4,348,000				4,348,000
12 months or more	[4]	3,866,000				3,866,000
Total	[4]	$ 8,214,000				$ 8,214,000
Number of Securities
Less than 12 Months | Security	[4]	35				35
12 Months or more | Security	[4]	11				11
Legacy Non-Agency MBS
MBS and CRT securities Interest Income
Coupon interest						$ 183,349,000	212,073,000		253,560,000
Effective yield adjustment	[5]					91,003,000	103,491,000		73,189,000
Interest income						274,352,000	315,564,000		326,749,000
CRT securities
Fair Value
Less than 12 months	[6]	$ 137,585,000				137,585,000
12 months or more	[6]	4,525,000				4,525,000
Total	[6]	142,110,000				142,110,000
Unrealized Losses
Less than 12 months	[6]	2,672,000				2,672,000
12 months or more	[6]	475,000				475,000
Total	[6]	$ 3,147,000				$ 3,147,000
Number of Securities
Less than 12 Months | Security	[6]	33				33
12 Months or more | Security	[6]	1				1
MBS and CRT securities Interest Income
Coupon interest						$ 5,844,000	665,000		0
Effective yield adjustment	[5]					728,000	107,000		0
Interest income						6,572,000	772,000		0
RPL/NPL MBS
MBS and CRT securities Interest Income
Coupon interest						87,429,000	898,000		21,000
Effective yield adjustment	[5]					1,789,000	(132,000)		0
Interest income						$ 89,218,000	$ 766,000		$ 21,000

[1] Includes amortization of premium paid net of accretion of purchase discount. For Agency MBS, interest income is recorded at an effective yield, which reflects net premium amortization based on actual prepayment activity.

[2] The Company reallocated $218,000 of purchase discount designated as Credit Reserve to accretable purchase discount on Non-Agency MBS underlying Linked Transactions for the year ended December 31, 2014.

[3] Together with coupon interest, accretable purchase discount is recognized as interest income over the life of the security.

[4] Based on management’s current estimates of future principal cash flows expected to be received.

[5] The effective yield adjustment is the difference between the net income calculated using the net yield, which is based on management’s estimates of the amount and timing of future cash flows, less the current coupon yield.

[6] Amounts disclosed at December 31, 2015 includes CRT securities with a fair value of $54.1 million for which the fair value option has been elected. Such securities have unrealized losses of $555,000 at December 31, 2015.