Investments in RMBS, Unrealized Loss Positions (Details) $ in Thousands		12 Months Ended
		Dec. 31, 2019 USD ($) Security	Dec. 31, 2018 USD ($) Security
RMBS, Unrealized Loss Positions [Abstract]
Carrying value		$ 2,508,360	$ 1,770,110
RMBS [Member]
RMBS, Unrealized Loss Positions [Abstract]
Original face value		2,880,949	2,042,097
Book value		2,466,828	1,808,420
Gross unrealized losses		(936)	(43,916)
Carrying value	[1]	$ 2,508,360	$ 1,770,110
Number of securities | Security		327	247
Weighted average coupon		3.85%	3.98%
Weighted average yield	[2]	3.72%	3.82%
Weighted average maturity	[3]	26 years	25 years
RMBS [Member] | Unrealized Loss Positions [Member]
RMBS, Unrealized Loss Positions [Abstract]
Original face value		$ 224,934	$ 1,769,106
Book value		186,969	1,545,732
Gross unrealized losses		(936)	(43,916)
Carrying value	[1]	$ 186,033	$ 1,501,816
Number of securities | Security		28	209
Weighted average coupon		3.74%	3.85%
Weighted average yield	[2]	3.54%	3.68%
Weighted average maturity	[4]	26 years	25 years
RMBS [Member] | Less than Twelve Months [Member] | Unrealized Loss Positions [Member]
RMBS, Unrealized Loss Positions [Abstract]
Original face value		$ 55,588	$ 256,937
Book value		55,429	224,617
Gross unrealized losses		(105)	(1,563)
Carrying value	[1]	$ 55,324	$ 223,054
Number of securities | Security		5	28
Weighted average rating	[5]
Weighted average coupon		3.70%	4.26%
Weighted average yield	[2]	3.53%	4.14%
Weighted average maturity	[4]	29 years	24 years
RMBS [Member] | Twelve or More Months [Member] | Unrealized Loss Positions [Member]
RMBS, Unrealized Loss Positions [Abstract]
Original face value		$ 169,346	$ 1,512,169
Book value		131,540	1,321,115
Gross unrealized losses		(831)	(42,353)
Carrying value	[1]	$ 130,709	$ 1,278,762
Number of securities | Security		23	181
Weighted average rating	[5]
Weighted average coupon		3.76%	3.78%
Weighted average yield	[2]	3.54%	3.60%
Weighted average maturity	[4]	25 years	25 years

[1] See Note 9 regarding the estimation of fair value, which approximates carrying value for all securities.

[2] The weighted average yield is based on the most recent gross monthly interest income, which is then annualized and divided by the book value of settled securities.

[3] The weighted average maturity is based on the timing of expected principal reduction on the assets.

[4] The weighted average maturity is based on the timing of expected principal reduction on the assets. Except for the security for which the Company has recognized OTTI, the Company does not intend to sell the investments and it is not more likely than not that the Company will be required to sell the investments before recovery of their amortized cost bases which may be maturity.

[5] The Company used an implied AAA rating for the Agency RMBS. CMOs issued by Fannie Mae or Freddie Mac consist of loss share securities, approximately half of which, by UPB, are unrated or rated below investment grade at December 31, 2019 by at least one NRSRO. Private label securities are rated investment grade or better by at least one NRSRO as of December 31, 2019.