Investments in RMBS, Unrealized Loss Positions (Details) $ in Thousands		12 Months Ended
		Dec. 31, 2018 USD ($) Security	Dec. 31, 2017 USD ($) Security
RMBS, Unrealized Loss Positions [Abstract]
Carrying value		$ 1,770,110	$ 1,840,912
RMBS [Member]
RMBS, Unrealized Loss Positions [Abstract]
Original face value		2,042,097	1,961,352
Book value		1,808,420	1,843,855
Gross unrealized losses		(43,916)	(11,577)
Carrying value	[1]	$ 1,770,110	$ 1,840,912
Number of securities | Security		247	238
Weighted average coupon		3.98%	3.86%
Weighted average yield	[2]	3.82%	3.66%
Weighted average maturity	[3]	25 years	25 years
RMBS [Member] | Unrealized Loss Positions [Member]
RMBS, Unrealized Loss Positions [Abstract]
Original face value		$ 24,355	$ 1,350,769
Book value		1,545,732	1,294,951
Gross unrealized losses		(43,916)	(11,577)
Carrying value	[1]	$ 1,501,816	$ 1,283,374
Number of securities | Security		209	156
Weighted average coupon		3.85%	3.76%
Weighted average yield	[2]	3.68%	3.58%
Weighted average maturity	[4]	25 years	26 years
RMBS [Member] | Less than Twelve Months [Member] | Unrealized Loss Positions [Member]
RMBS, Unrealized Loss Positions [Abstract]
Original face value		$ 13,909	$ 1,026,911
Book value		224,617	1,005,352
Gross unrealized losses		(1,563)	(5,378)
Carrying value	[1]	$ 223,054	$ 999,974
Number of securities | Security		28	111
Weighted average rating	[5]
Weighted average coupon		4.26%	3.81%
Weighted average yield	[2]	4.14%	3.63%
Weighted average maturity	[4]	24 years	26 years
RMBS [Member] | Twelve or More Months [Member] | Unrealized Loss Positions [Member]
RMBS, Unrealized Loss Positions [Abstract]
Original face value		$ 10,446	$ 323,858
Book value		1,321,115	289,599
Gross unrealized losses		(42,353)	(6,199)
Carrying value	[1]	$ 1,278,762	$ 283,400
Number of securities | Security		181	45
Weighted average rating	[5]
Weighted average coupon		3.78%	3.61%
Weighted average yield	[2]	3.60%	3.40%
Weighted average maturity	[4]	25 years	25 years

[1] See Note 9 regarding the estimation of fair value, which approximates carrying value for all securities.

[2] The weighted average yield is based on the most recent gross monthly interest income, which is then annualized and divided by the book value of settled securities.

[3] The weighted average maturity is based on the timing of expected principal reduction on the assets.

[4] The weighted average maturity is based on the timing of expected principal reduction on the assets. Except for the security for which the Company has recognized OTTI, the Company does not intend to sell the investments and it is not more likely than not that the Company will be required to sell the investments before recovery of their amortized cost bases which may be maturity.

[5] The Company used an implied AAA rating for the Agency RMBS. CMOs issued by Fannie Mae or Freddie Mac consist of loss share securities, the majority of which, by UPB, are unrated or rated below investment grade at December 31, 2018 by at least one NRSRO. Private label securities are rated investment grade or better by at least one NRSRO as of December 31, 2018.