Investments in RMBS, Unrealized Loss Positions (Details) $ in Thousands		9 Months Ended	12 Months Ended
		Sep. 30, 2019 USD ($) Security	Dec. 31, 2018 USD ($) Security
RMBS, Unrealized Loss Positions [Abstract]
Carrying value		$ 2,483,237	$ 1,770,110
RMBS [Member]
RMBS, Unrealized Loss Positions [Abstract]
Original face value		2,805,818	2,042,097
Book value		2,448,451	1,808,420
Gross unrealized losses		(2,146)	(43,916)
Carrying value	[1]	$ 2,483,237	$ 1,770,110
Number of securities | Security		325	247
Weighted average coupon		3.89%	3.98%
Weighted average yield	[2]	3.77%	3.82%
Weighted average maturity	[3]	26 years	25 years
RMBS [Member] | Unrealized Loss Positions [Member]
RMBS, Unrealized Loss Positions [Abstract]
Original face value		$ 431,151	$ 1,769,106
Book value		369,339	1,545,732
Gross unrealized losses		(2,146)	(43,916)
Carrying value	[1]	$ 367,193	$ 1,501,816
Number of securities | Security		55	209
Weighted average coupon		3.71%	3.85%
Weighted average yield	[2]	3.51%	3.68%
Weighted average maturity	[4]	26 years	25 years
RMBS [Member] | Less than Twelve Months [Member] | Unrealized Loss Positions [Member]
RMBS, Unrealized Loss Positions [Abstract]
Original face value		$ 197,867	$ 256,937
Book value		183,984	224,617
Gross unrealized losses		(355)	(1,563)
Carrying value	[1]	$ 183,629	$ 223,054
Number of securities | Security		22	28
Weighted average rating	[5]
Weighted average coupon		3.70%	4.26%
Weighted average yield	[2]	3.52%	4.14%
Weighted average maturity	[4]	28 years	24 years
RMBS [Member] | Twelve or More Months [Member] | Unrealized Loss Positions [Member]
RMBS, Unrealized Loss Positions [Abstract]
Original face value		$ 233,284	$ 1,512,169
Book value		185,355	1,321,115
Gross unrealized losses		(1,791)	(42,353)
Carrying value	[1]	$ 183,564	$ 1,278,762
Number of securities | Security		33	181
Weighted average rating	[5]
Weighted average coupon		3.72%	3.78%
Weighted average yield	[2]	3.50%	3.60%
Weighted average maturity	[4]	24 years	25 years

[1] See Note 9 regarding the estimation of fair value, which approximates carrying value for all securities.

[2] The weighted average yield is based on the most recent gross monthly interest income, which is then annualized and divided by the book value of settled securities.

[3] The weighted average maturity is based on the timing of expected principal reduction on the assets.

[4] The weighted average maturity is based on the timing of expected principal reduction on the assets. Except for the security for which the Company has recognized OTTI, the Company does not intend to sell the investments and it is not more likely than not that the Company will be required to sell the investments before recovery of their amortized cost bases which may be maturity.

[5] The Company used an implied AAA rating for the Agency RMBS. CMOs issued by Fannie Mae or Freddie Mac consist of loss share securities, the majority of which, by UPB, are unrated or rated below investment grade at September 30, 2019 by at least one NRSRO. The Company's private label securities are rated investment grade or better by at least one NRSRO as of September 30, 2019.