Fair Value of Financial Instruments (Details) - USD ($) $ in Thousands	Dec. 31, 2020	Dec. 31, 2019
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	$ 11,342	$ 4,290
Derivative liabilities	1,634	974
Mortgage loans held for investment (includes $6,264 and $8,857 at fair value, respectively); see Note 3	6,264	8,857
Fair Value, Measurements, Recurring
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities	2,596,255	5,188,163
Assets, Fair Value Disclosure	2,613,861	5,201,310
Financial Liabilities Fair Value Disclosure	1,634	974
Mortgage loans held for investment (includes $6,264 and $8,857 at fair value, respectively); see Note 3	6,264	8,857	[1]
Fair Value, Measurements, Recurring | Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities	0	0
Assets, Fair Value Disclosure	1,094	2,883
Financial Liabilities Fair Value Disclosure	1,527	0
Mortgage loans held for investment (includes $6,264 and $8,857 at fair value, respectively); see Note 3	0	0
Fair Value, Measurements, Recurring | Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities	2,594,980	5,186,473
Assets, Fair Value Disclosure	2,605,228	5,187,880
Financial Liabilities Fair Value Disclosure	107	974
Mortgage loans held for investment (includes $6,264 and $8,857 at fair value, respectively); see Note 3	0	0
Fair Value, Measurements, Recurring | Level 3
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities	1,275	1,690
Assets, Fair Value Disclosure	7,539	10,547
Financial Liabilities Fair Value Disclosure	0	0
Mortgage loans held for investment (includes $6,264 and $8,857 at fair value, respectively); see Note 3	6,264	8,857	[1]
Fair Value, Measurements, Recurring | Options on U.S. Treasury futures
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	1,094	2,883
Fair Value, Measurements, Recurring | Options on U.S. Treasury futures | Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	1,094	2,883
Fair Value, Measurements, Recurring | Options on U.S. Treasury futures | Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	0	0
Fair Value, Measurements, Recurring | Options on U.S. Treasury futures | Level 3
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	0	0
Fair Value, Measurements, Recurring | Interest rate swaption
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	1,360	573
Derivative liabilities	107	0
Fair Value, Measurements, Recurring | Interest rate swaption | Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	0	0
Derivative liabilities	0	0
Fair Value, Measurements, Recurring | Interest rate swaption | Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	1,360	573
Derivative liabilities	107	0
Fair Value, Measurements, Recurring | Interest rate swaption | Level 3
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	0	0
Derivative liabilities	0	0
Fair Value, Measurements, Recurring | TBA securities | Short position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	0	974
Fair Value, Measurements, Recurring | TBA securities | Long position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	8,888	834
Fair Value, Measurements, Recurring | TBA securities | Level 1 | Short position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	0	0
Fair Value, Measurements, Recurring | TBA securities | Level 1 | Long position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	0	0
Fair Value, Measurements, Recurring | TBA securities | Level 2 | Short position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	0	974
Fair Value, Measurements, Recurring | TBA securities | Level 2 | Long position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	8,888	834
Fair Value, Measurements, Recurring | TBA securities | Level 3 | Short position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	0	0
Fair Value, Measurements, Recurring | TBA securities | Level 3 | Long position
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative asset	0	0
Fair Value, Measurements, Recurring | Futures
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	1,527	0
Fair Value, Measurements, Recurring | Futures | Level 1
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	1,527	0
Fair Value, Measurements, Recurring | Futures | Level 2
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	0	0
Fair Value, Measurements, Recurring | Futures | Level 3
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative liabilities	$ 0	$ 0

[1] Mortgage loans held for investment were carried at amortized cost of $9,405 on the Company’s consolidated balance sheet as of December 31, 2019.