Fair Value of Financial Instruments (Details) - USD ($) $ in Thousands	Mar. 31, 2020	Dec. 31, 2019
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	$ 3,031	$ 4,290
Derivative liabilities	13,640	974
Fair Value, Measurements, Recurring
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities	3,719,706	5,188,163
Assets, Fair Value Disclosure	3,730,659	5,192,453
Financial Liabilities Fair Value Disclosure	13,640	974
Loans Receivable, Fair Value Disclosure	7,922	0
Fair Value, Measurements, Recurring | Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities	0	0
Assets, Fair Value Disclosure	3,031	2,883
Financial Liabilities Fair Value Disclosure	8,438	0
Loans Receivable, Fair Value Disclosure	0	0
Fair Value, Measurements, Recurring | Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities	3,718,144	5,186,473
Assets, Fair Value Disclosure	3,718,144	5,187,880
Financial Liabilities Fair Value Disclosure	5,202	974
Loans Receivable, Fair Value Disclosure	0	0
Fair Value, Measurements, Recurring | Level 3
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities	1,562	1,690
Assets, Fair Value Disclosure	9,484	1,690
Financial Liabilities Fair Value Disclosure	0	0
Loans Receivable, Fair Value Disclosure	7,922	0
Fair Value, Measurements, Recurring | Options on U.S. Treasury futures
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	3,031	2,883
Fair Value, Measurements, Recurring | Options on U.S. Treasury futures | Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	3,031	2,883
Fair Value, Measurements, Recurring | Options on U.S. Treasury futures | Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	0	0
Fair Value, Measurements, Recurring | Options on U.S. Treasury futures | Level 3
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	0	0
Fair Value, Measurements, Recurring | Interest rate swaption
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	0	573
Fair Value, Measurements, Recurring | Interest rate swaption | Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	0	0
Fair Value, Measurements, Recurring | Interest rate swaption | Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	0	573
Fair Value, Measurements, Recurring | Interest rate swaption | Level 3
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	0	0
Fair Value, Measurements, Recurring | TBA securities | Short position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	5,202	974
Fair Value, Measurements, Recurring | TBA securities | Long position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	0	834
Fair Value, Measurements, Recurring | TBA securities | Level 1 | Short position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	0	0
Fair Value, Measurements, Recurring | TBA securities | Level 1 | Long position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	0	0
Fair Value, Measurements, Recurring | TBA securities | Level 2 | Short position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	5,202	974
Fair Value, Measurements, Recurring | TBA securities | Level 2 | Long position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	0	834
Fair Value, Measurements, Recurring | TBA securities | Level 3 | Short position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	0	0
Fair Value, Measurements, Recurring | TBA securities | Level 3 | Long position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	0	0
Fair Value, Measurements, Recurring | Futures
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	8,438	0
Fair Value, Measurements, Recurring | Futures | Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	8,438	0
Fair Value, Measurements, Recurring | Futures | Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	0	0
Fair Value, Measurements, Recurring | Futures | Level 3
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	$ 0	$ 0