Fair Value of Financial Instruments Fair Value on a Recurring Basis (Details) - USD ($) $ in Thousands	Jun. 30, 2017	Dec. 31, 2016
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities	$ 2,864,026	$ 3,212,084
Derivative Instruments Not Designated as Hedging Instruments, Asset, at Fair Value	267	28,534	[1]
Derivative asset	267	28,534
Total assets carried at fair value	2,864,293	3,240,618
Derivative liabilities	1,686	6,922
Total liabilities carried at fair value	1,686	6,922
Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities	0	0
Total assets carried at fair value	0	0
Total liabilities carried at fair value	0	0
Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities	2,856,229	3,201,157
Total assets carried at fair value	2,856,496	3,229,691
Total liabilities carried at fair value	1,686	6,922
Level 3
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities	7,797	10,927
Total assets carried at fair value	7,797	10,927
Total liabilities carried at fair value	0	0
TBA security
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	1,668
TBA security | Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	0
TBA security | Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	1,668
TBA security | Level 3
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	0
Interest Rate Swap
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset		28,534
Derivative liabilities	18	6,922
Interest Rate Swap | Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	0	0
Derivative liabilities	0	0
Interest Rate Swap | Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	267	28,534
Derivative liabilities	18	6,922
Interest Rate Swap | Level 3
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	0	0
Derivative liabilities	0	0
Interest Rate Swap
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative Instruments Not Designated as Hedging Instruments, Asset, at Fair Value	$ 267	$ 28,534	[1]

[1] Refer to Note 1 regarding information on a change in the CME rulebook. Amounts reported on the consolidated balance sheet as of June 30, 2017 for its interest rate swaps reflect the netting of the derivative asset or liability with the related collateral received or posted, respectively. The net amounts comparable to June 30, 2017 for the derivative asset and derivative liabilities as of December 31, 2016 were $104 and $(576), respectively.