Fair Value of Financial Instruments (Details) - USD ($) $ in Thousands	Jun. 30, 2020	Dec. 31, 2019
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	$ 7,926	$ 4,290
Derivative liabilities	4,208	974
Fair Value, Measurements, Recurring
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities	3,496,925	5,188,163
Assets, Fair Value Disclosure	3,512,217	5,192,453
Financial Liabilities Fair Value Disclosure	4,208	974
Loans Receivable, Fair Value Disclosure	7,366	0
Fair Value, Measurements, Recurring | Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities	0	0
Assets, Fair Value Disclosure	3,168	2,883
Financial Liabilities Fair Value Disclosure	4,208	0
Loans Receivable, Fair Value Disclosure	0	0
Fair Value, Measurements, Recurring | Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities	3,495,435	5,186,473
Assets, Fair Value Disclosure	3,500,193	5,187,880
Financial Liabilities Fair Value Disclosure	0	974
Loans Receivable, Fair Value Disclosure	0	0
Fair Value, Measurements, Recurring | Level 3
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities	1,490	1,690
Assets, Fair Value Disclosure	8,856	1,690
Financial Liabilities Fair Value Disclosure	0	0
Loans Receivable, Fair Value Disclosure	7,366	0
Fair Value, Measurements, Recurring | Options on U.S. Treasury futures
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	3,168	2,883
Fair Value, Measurements, Recurring | Options on U.S. Treasury futures | Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	3,168	2,883
Fair Value, Measurements, Recurring | Options on U.S. Treasury futures | Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	0	0
Fair Value, Measurements, Recurring | Options on U.S. Treasury futures | Level 3
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	0	0
Fair Value, Measurements, Recurring | Interest rate swaption
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset		573
Fair Value, Measurements, Recurring | Interest rate swaption | Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset		0
Fair Value, Measurements, Recurring | Interest rate swaption | Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset		573
Fair Value, Measurements, Recurring | Interest rate swaption | Level 3
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset		0
Fair Value, Measurements, Recurring | TBA securities | Short position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities		974
Fair Value, Measurements, Recurring | TBA securities | Long position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	4,758	834
Fair Value, Measurements, Recurring | TBA securities | Level 1 | Short position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities		0
Fair Value, Measurements, Recurring | TBA securities | Level 1 | Long position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	0	0
Fair Value, Measurements, Recurring | TBA securities | Level 2 | Short position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities		974
Fair Value, Measurements, Recurring | TBA securities | Level 2 | Long position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	4,758	834
Fair Value, Measurements, Recurring | TBA securities | Level 3 | Short position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities		0
Fair Value, Measurements, Recurring | TBA securities | Level 3 | Long position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	0	0
Fair Value, Measurements, Recurring | Futures
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	4,208	0
Fair Value, Measurements, Recurring | Futures | Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	4,208	0
Fair Value, Measurements, Recurring | Futures | Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	0	0
Fair Value, Measurements, Recurring | Futures | Level 3
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	$ 0	$ 0