Fair Value of Financial Instruments (Details) - USD ($) $ in Thousands	Sep. 30, 2020	Dec. 31, 2019
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	$ 4,266	$ 4,290
Derivative liabilities	5,164	974
Mortgage loans held for investment (includes 6,920,907 and $8,857 at fair value, respectively); see Note 1	6,921	8,857
Fair Value, Measurements, Recurring
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities	2,995,660	5,188,163
Assets, Fair Value Disclosure	3,006,847	5,201,310
Financial Liabilities Fair Value Disclosure	5,164	974
Mortgage loans held for investment (includes 6,920,907 and $8,857 at fair value, respectively); see Note 1	6,921	8,857	[1]
Fair Value, Measurements, Recurring | Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities	0	0
Assets, Fair Value Disclosure	386	2,883
Financial Liabilities Fair Value Disclosure	5,164	0
Mortgage loans held for investment (includes 6,920,907 and $8,857 at fair value, respectively); see Note 1	0	0
Fair Value, Measurements, Recurring | Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities	2,994,262	5,186,473
Assets, Fair Value Disclosure	2,998,142	5,187,880
Financial Liabilities Fair Value Disclosure	0	974
Mortgage loans held for investment (includes 6,920,907 and $8,857 at fair value, respectively); see Note 1	0	0
Fair Value, Measurements, Recurring | Level 3
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities	1,398	1,690
Assets, Fair Value Disclosure	8,319	10,547
Financial Liabilities Fair Value Disclosure	0	0
Mortgage loans held for investment (includes 6,920,907 and $8,857 at fair value, respectively); see Note 1	6,921	8,857	[1]
Fair Value, Measurements, Recurring | Options on U.S. Treasury futures
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	386	2,883
Fair Value, Measurements, Recurring | Options on U.S. Treasury futures | Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	386	2,883
Fair Value, Measurements, Recurring | Options on U.S. Treasury futures | Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	0	0
Fair Value, Measurements, Recurring | Options on U.S. Treasury futures | Level 3
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	0	0
Fair Value, Measurements, Recurring | Interest rate swaption
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	411	573
Fair Value, Measurements, Recurring | Interest rate swaption | Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	0	0
Fair Value, Measurements, Recurring | Interest rate swaption | Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	411	573
Fair Value, Measurements, Recurring | Interest rate swaption | Level 3
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	0	0
Fair Value, Measurements, Recurring | TBA securities | Short position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	0	974
Fair Value, Measurements, Recurring | TBA securities | Long position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	3,469	834
Fair Value, Measurements, Recurring | TBA securities | Level 1 | Short position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	0	0
Fair Value, Measurements, Recurring | TBA securities | Level 1 | Long position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	0	0
Fair Value, Measurements, Recurring | TBA securities | Level 2 | Short position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	0	974
Fair Value, Measurements, Recurring | TBA securities | Level 2 | Long position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	3,469	834
Fair Value, Measurements, Recurring | TBA securities | Level 3 | Short position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	0	0
Fair Value, Measurements, Recurring | TBA securities | Level 3 | Long position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	0	0
Fair Value, Measurements, Recurring | Futures
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	5,164	0
Fair Value, Measurements, Recurring | Futures | Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	5,164	0
Fair Value, Measurements, Recurring | Futures | Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	0	0
Fair Value, Measurements, Recurring | Futures | Level 3
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	$ 0	$ 0

[1] Mortgage loans held for investment were carried at amortized cost of $9,405 on the Company’s consolidated balance sheet as of December 31, 2019.