Repurchase Agreements and Other Advances (Details 2) - USD ($) $ in Thousands	Jun. 30, 2016	Dec. 31, 2015
Information about MBS [Line Items]
Less unamortized debt issuance costs	$ 450	$ 1,280
Total repurchase agreements less debt issuance costs	8,493,087	7,887,622
Amounts related to repurchase agreements not included in offsetting disclosure in Note 11	0
Repurchase Agreement Borrowings
Information about MBS [Line Items]
Balance of repurchase agreements	$ 8,493,537	$ 7,888,902
Weighted Average Interest Rate	1.47%	1.48%
Repurchase Agreement Borrowings | Overnight
Information about MBS [Line Items]
Balance of repurchase agreements	$ 0
Weighted Average Interest Rate	0.00%
Repurchase Agreement Borrowings | Within 30 days
Information about MBS [Line Items]
Balance of repurchase agreements	$ 6,214,735
Weighted Average Interest Rate	1.20%
Repurchase Agreement Borrowings | Over 30 days to 3 months
Information about MBS [Line Items]
Balance of repurchase agreements	$ 1,252,273
Weighted Average Interest Rate	2.00%
Repurchase Agreement Borrowings | Over 3 months to 12 months
Information about MBS [Line Items]
Balance of repurchase agreements	$ 859,667
Weighted Average Interest Rate	2.26%
Repurchase Agreement Borrowings | Over 12 months
Information about MBS [Line Items]
Balance of repurchase agreements	$ 166,862
Weighted Average Interest Rate	3.16%
Repurchase Agreement Borrowings | Agency MBS
Information about MBS [Line Items]
Balance of repurchase agreements	$ 3,276,823	$ 2,727,542
Repurchase Agreement Borrowings | Agency MBS | Overnight
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | Agency MBS | Within 30 days
Information about MBS [Line Items]
Balance of repurchase agreements	3,190,594
Repurchase Agreement Borrowings | Agency MBS | Over 30 days to 3 months
Information about MBS [Line Items]
Balance of repurchase agreements	86,229
Repurchase Agreement Borrowings | Agency MBS | Over 3 months to 12 months
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | Agency MBS | Over 12 months
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | Legacy Non-Agency MBS
Information about MBS [Line Items]
Balance of repurchase agreements	1,865,974
Repurchase Agreement Borrowings | Legacy Non-Agency MBS | Overnight
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | Legacy Non-Agency MBS | Within 30 days
Information about MBS [Line Items]
Balance of repurchase agreements	933,213
Repurchase Agreement Borrowings | Legacy Non-Agency MBS | Over 30 days to 3 months
Information about MBS [Line Items]
Balance of repurchase agreements	570,918
Repurchase Agreement Borrowings | Legacy Non-Agency MBS | Over 3 months to 12 months
Information about MBS [Line Items]
Balance of repurchase agreements	361,843
Repurchase Agreement Borrowings | Legacy Non-Agency MBS | Over 12 months
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | RPL/NPL MBS
Information about MBS [Line Items]
Balance of repurchase agreements	2,069,976	2,080,163
Repurchase Agreement Borrowings | RPL/NPL MBS | Overnight
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | RPL/NPL MBS | Within 30 days
Information about MBS [Line Items]
Balance of repurchase agreements	1,426,126
Repurchase Agreement Borrowings | RPL/NPL MBS | Over 30 days to 3 months
Information about MBS [Line Items]
Balance of repurchase agreements	428,483
Repurchase Agreement Borrowings | RPL/NPL MBS | Over 3 months to 12 months
Information about MBS [Line Items]
Balance of repurchase agreements	215,367
Repurchase Agreement Borrowings | RPL/NPL MBS | Over 12 months
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | U.S. Treasuries
Information about MBS [Line Items]
Balance of repurchase agreements	505,751	504,760
Repurchase Agreement Borrowings | U.S. Treasuries | Overnight
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | U.S. Treasuries | Within 30 days
Information about MBS [Line Items]
Balance of repurchase agreements	505,751
Repurchase Agreement Borrowings | U.S. Treasuries | Over 30 days to 3 months
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | U.S. Treasuries | Over 3 months to 12 months
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | U.S. Treasuries | Over 12 months
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | CRT securities
Information about MBS [Line Items]
Balance of repurchase agreements	190,973	128,465
Repurchase Agreement Borrowings | CRT securities | Overnight
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | CRT securities | Within 30 days
Information about MBS [Line Items]
Balance of repurchase agreements	159,051
Repurchase Agreement Borrowings | CRT securities | Over 30 days to 3 months
Information about MBS [Line Items]
Balance of repurchase agreements	31,922
Repurchase Agreement Borrowings | CRT securities | Over 3 months to 12 months
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | CRT securities | Over 12 months
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | Residential Whole Loans
Information about MBS [Line Items]
Balance of repurchase agreements	584,040	487,750
Repurchase Agreement Borrowings | Residential Whole Loans | Overnight
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | Residential Whole Loans | Within 30 days
Information about MBS [Line Items]
Balance of repurchase agreements	0
Repurchase Agreement Borrowings | Residential Whole Loans | Over 30 days to 3 months
Information about MBS [Line Items]
Balance of repurchase agreements	134,721
Repurchase Agreement Borrowings | Residential Whole Loans | Over 3 months to 12 months
Information about MBS [Line Items]
Balance of repurchase agreements	282,457
Repurchase Agreement Borrowings | Residential Whole Loans | Over 12 months
Information about MBS [Line Items]
Balance of repurchase agreements	166,862
Repurchase Agreement Borrowings | Within 30 days
Information about MBS [Line Items]
Balance of repurchase agreements	$ 7,340,126	$ 7,054,483
Weighted Average Interest Rate	1.39%	1.44%
Repurchase Agreement Borrowings | Over 30 days to 3 months
Information about MBS [Line Items]
Balance of repurchase agreements	$ 896,242	$ 734,955
Weighted Average Interest Rate	1.92%	1.79%
Repurchase Agreement Borrowings | Over 3 months to 12 months
Information about MBS [Line Items]
Balance of repurchase agreements	$ 257,169	$ 99,464
Weighted Average Interest Rate	2.03%	2.36%