Other Assets - Derivative Instruments (Details 2) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2016 USD ($)	Sep. 30, 2016 USD ($)	Jun. 30, 2016 USD ($)	Mar. 31, 2016 USD ($)	Dec. 31, 2015 USD ($)	Sep. 30, 2015 USD ($)	Jun. 30, 2015 USD ($)	Mar. 31, 2015 USD ($)	Dec. 31, 2016 USD ($) derivative	Dec. 31, 2015 USD ($) derivative	Dec. 31, 2014 USD ($)	Dec. 31, 2016 USD ($)	Dec. 31, 2015 USD ($)
Derivative [Line Items]
Assets Pledged												$ 10,620,406	$ 11,410,397
Number of terminated derivatives | derivative									0	0
Net unrealized gains (losses)	$ (46,721)			$ (69,399)	$ (69,399)			$ (59,062)	$ (69,399)	$ (59,062)	$ (15,217)	(46,721)	(69,399)
Interest expense attributable to Swaps	49,868	$ 48,167	$ 47,720	47,600	46,456	$ 43,703	$ 42,849	43,940	193,355	176,948	159,808
AOCI from derivative hedging instruments:
Balance at beginning of period				$ (69,399)				$ (59,062)	(69,399)	(59,062)	(15,217)
Unrealized gain/(loss) on Swaps, net									22,678	(10,337)	(44,292)
Balance at end of period	(46,721)				$ (69,399)				$ (46,721)	(69,399)	(59,062)
Interest Rate Contract
Derivative [Line Items]
Aggregate fair value of assets needed to immediately settle												48,000
Associated accrued interest payable												1,000
Assets Pledged												86,317	109,142
Interest Rate Contract | Agency MBS, at fair value
Derivative [Line Items]
Assets Pledged												32,468	38,569
Interest Rate Contract | Restricted cash
Derivative [Line Items]
Assets Pledged												53,849	70,573
Swaps, at fair value
Derivative [Line Items]
Aggregate notional amount of derivatives												$ 2,900,000	$ 3,050,000
Average maturity term of swaps									35 months
Weighted average fixed-pay rate												1.87%	1.82%
Weighted Average Variable Interest Rate												0.72%	0.34%
Interest expense attributable to Swaps									$ 40,898	$ 53,759	$ 69,842
Weighted average Swap rate paid									1.82%	1.86%	1.93%
Weighted average Swap rate received									0.48%	0.19%	0.16%
Swaps, at fair value | Within 30 days
Derivative [Line Items]
Aggregate notional amount of derivatives												$ 0	$ 50,000
Weighted average fixed-pay rate												0.00%	2.13%
Weighted Average Variable Interest Rate												0.00%	0.42%
Swaps, at fair value | Over 30 days to 3 months
Derivative [Line Items]
Aggregate notional amount of derivatives												$ 50,000	$ 0
Weighted average fixed-pay rate												0.67%	0.00%
Weighted Average Variable Interest Rate												0.64%	0.00%
Swaps, at fair value | Over 3 months to 6 months
Derivative [Line Items]
Aggregate notional amount of derivatives												$ 300,000	$ 0
Weighted average fixed-pay rate												0.57%	0.00%
Weighted Average Variable Interest Rate												0.66%	0.00%
Swaps, at fair value | Over 6 months to 12 months
Derivative [Line Items]
Aggregate notional amount of derivatives												$ 0	$ 100,000
Weighted average fixed-pay rate												0.00%	0.48%
Weighted Average Variable Interest Rate												0.00%	0.32%
Swaps, at fair value | Over 12 months to 24 months
Derivative [Line Items]
Aggregate notional amount of derivatives												$ 550,000	$ 350,000
Weighted average fixed-pay rate												1.49%	0.58%
Weighted Average Variable Interest Rate												0.71%	0.27%
Swaps, at fair value | Over 24 months to 36 months
Derivative [Line Items]
Aggregate notional amount of derivatives												$ 200,000	$ 550,000
Weighted average fixed-pay rate												1.71%	1.49%
Weighted Average Variable Interest Rate												0.76%	0.32%
Swaps, at fair value | Over 36 months to 48 months
Derivative [Line Items]
Aggregate notional amount of derivatives												$ 1,500,000	$ 200,000
Weighted average fixed-pay rate												2.22%	1.71%
Weighted Average Variable Interest Rate												0.74%	0.42%
Swaps, at fair value | Over 48 months to 60 months
Derivative [Line Items]
Aggregate notional amount of derivatives												$ 200,000	$ 1,500,000
Weighted average fixed-pay rate												2.20%	2.22%
Weighted Average Variable Interest Rate												0.75%	0.36%
Swaps, at fair value | Over 60 months to 72 months
Derivative [Line Items]
Aggregate notional amount of derivatives												$ 0	$ 200,000
Weighted average fixed-pay rate												0.00%	2.20%
Weighted Average Variable Interest Rate												0.00%	0.30%
Swaps, at fair value | Over 72 months to 84 months
Derivative [Line Items]
Aggregate notional amount of derivatives												$ 100,000	$ 0
Weighted average fixed-pay rate												2.75%	0.00%
Weighted Average Variable Interest Rate												0.74%	0.00%
Swaps, at fair value | Over 84 months
Derivative [Line Items]
Aggregate notional amount of derivatives												$ 0	$ 100,000
Weighted average fixed-pay rate												0.00%	2.75%
Weighted Average Variable Interest Rate												0.00%	0.40%
Swaps, at fair value | Hedging
Derivative [Line Items]
Net unrealized gains (losses)	(46,700)								$ (46,700)			$ (46,700)
AOCI from derivative hedging instruments:
Balance at end of period	$ (46,700)								$ (46,700)
Swaps, at fair value | Minimum | LIBOR
Derivative [Line Items]
Derivative, variable interest rate, term									1 month
Swaps, at fair value | Maximum
Derivative [Line Items]
Maximum maturity term of swaps									80 months
Swaps, at fair value | Maximum | LIBOR
Derivative [Line Items]
Derivative, variable interest rate, term									3 months
New Swaps:
Derivative [Line Items]
Aggregate notional amount									$ 0
Weighted average fixed-pay rate												0.00%
Number of new Swaps | derivative									0
Swaps amortized/expired:
Derivative [Line Items]
Aggregate notional amount									$ 150,000
Swaps amortized/expired: | YTD
Derivative [Line Items]
Weighted average fixed-pay rate												1.03%