Fair Value of Financial Instruments Fair Value on a Recurring Basis (Details) - USD ($) $ in Thousands	Dec. 31, 2016	Dec. 31, 2015
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities	$ 3,212,084	$ 3,493,701
Derivative Instruments Not Designated as Hedging Instruments, Asset, at Fair Value	28,534	7,835
Derivative assets	28,534	7,835
Derivative liabilities	6,922	41,205
Fair Value, Measurements, Recurring
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities	3,212,084	3,493,701
Derivative Instruments Not Designated as Hedging Instruments, Asset, at Fair Value	28,534
Derivative assets		7,835
Total assets carried at fair value	3,240,618	3,501,536
Derivative liabilities	6,922	41,205
Total liabilities carried at fair value	6,922	41,205
Fair Value, Measurements, Recurring | Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities	0	0
Derivative assets	0	0
Total assets carried at fair value	0	0
Derivative liabilities	0	29,097
Total liabilities carried at fair value	0	29,097
Fair Value, Measurements, Recurring | Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities	3,201,157	3,477,266
Derivative assets	28,534	7,835
Total assets carried at fair value	3,229,691	3,485,101
Derivative liabilities	6,922	12,108
Total liabilities carried at fair value	6,922	12,108
Fair Value, Measurements, Recurring | Level 3
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities	10,927	16,435
Derivative assets	0	0
Total assets carried at fair value	10,927	16,435
Derivative liabilities	0	0
Total liabilities carried at fair value	$ 0	$ 0