Investments in RMBS, Assets by Maturity (Details) $ in Thousands		12 Months Ended
		Dec. 31, 2018 USD ($) Security	Dec. 31, 2017 USD ($) Security
RMBS, Assets by Maturity [Abstract]
Carrying value		$ 1,770,110	$ 1,840,912
RMBS [Member]
RMBS, Assets by Maturity [Abstract]
Original face value		2,042,097	1,961,352
Book value		1,808,420	1,843,855
Gross unrealized gains		5,606	8,634
Gross unrealized losses		(43,916)	(11,577)
Carrying value	[1]	$ 1,770,110	$ 1,840,912
Number of securities | Security		247	238
Weighted average coupon		3.98%	3.86%
Weighted average yield	[2]	3.82%	3.66%
Weighted average maturity	[3]	25 years	25 years
RMBS [Member] | 5-10 Years [Member]
RMBS, Assets by Maturity [Abstract]
Original face value		$ 24,377	$ 16,069
Book value		15,100	15,483
Gross unrealized gains		731	324
Gross unrealized losses		(134)	(312)
Carrying value	[1]	$ 15,697	$ 15,495
Number of securities | Security		7	3
Weighted average rating	[4]
Weighted average coupon		4.97%	4.33%
Weighted average yield	[2]	4.93%	4.06%
Weighted average maturity	[3]	9 years	7 years
RMBS [Member] | Over 10 Years [Member]
RMBS, Assets by Maturity [Abstract]
Original face value		$ 2,017,720	$ 1,945,283
Book value		1,793,320	1,828,372
Gross unrealized gains		4,875	8,310
Gross unrealized losses		(43,782)	(11,265)
Carrying value	[1]	$ 1,754,413	$ 1,825,417
Number of securities | Security		240	235
Weighted average rating	[4]
Weighted average coupon		3.97%	3.85%
Weighted average yield	[2]	3.81%	3.65%
Weighted average maturity	[3]	25 years	26 years

[1] See Note 9 regarding the estimation of fair value, which approximates carrying value for all securities.

[2] The weighted average yield is based on the most recent gross monthly interest income, which is then annualized and divided by the book value of settled securities.

[3] The weighted average maturity is based on the timing of expected principal reduction on the assets.

[4] The Company used an implied AAA rating for the Agency RMBS. CMOs issued by Fannie Mae or Freddie Mac consist of loss share securities, the majority of which, by UPB, are unrated or rated below investment grade at December 31, 2018 by at least one NRSRO. Private label securities are rated investment grade or better by at least one NRSRO as of December 31, 2018.