Investments in RMBS, Assets by Maturity (Details) $ in Thousands		9 Months Ended	12 Months Ended
		Sep. 30, 2019 USD ($) Security	Dec. 31, 2018 USD ($) Security
RMBS, Assets by Maturity [Abstract]
Carrying value		$ 2,483,237	$ 1,770,110
RMBS [Member]
RMBS, Assets by Maturity [Abstract]
Original face value		2,805,818	2,042,097
Book value		2,448,451	1,808,420
Gross unrealized gains		36,932	5,606
Gross unrealized losses		(2,146)	(43,916)
Carrying value	[1]	$ 2,483,237	$ 1,770,110
Number of securities | Security		325	247
Weighted average coupon		3.89%	3.98%
Weighted average yield	[2]	3.77%	3.82%
Weighted average maturity	[3]	26 years	25 years
RMBS [Member] | 5-10 Years [Member]
RMBS, Assets by Maturity [Abstract]
Original face value		$ 54,918	$ 24,377
Book value		47,742	15,100
Gross unrealized gains		2,287	731
Gross unrealized losses		0	(134)
Carrying value	[1]	$ 50,029	$ 15,697
Number of securities | Security		14	7
Weighted average rating	[4]
Weighted average coupon		5.78%	4.97%
Weighted average yield	[2]	6.14%	4.93%
Weighted average maturity	[3]	9 years	9 years
RMBS [Member] | Over 10 Years [Member]
RMBS, Assets by Maturity [Abstract]
Original face value		$ 2,750,900	$ 2,017,720
Book value		2,400,709	1,793,320
Gross unrealized gains		34,645	4,875
Gross unrealized losses		(2,146)	(43,782)
Carrying value	[1]	$ 2,433,208	$ 1,754,413
Number of securities | Security		311	240
Weighted average rating	[4]
Weighted average coupon		3.85%	3.97%
Weighted average yield	[2]	3.72%	3.81%
Weighted average maturity	[3]	26 years	25 years

[1] See Note 9 regarding the estimation of fair value, which approximates carrying value for all securities.

[2] The weighted average yield is based on the most recent gross monthly interest income, which is then annualized and divided by the book value of settled securities.

[3] The weighted average maturity is based on the timing of expected principal reduction on the assets.

[4] The Company used an implied AAA rating for the Agency RMBS. CMOs issued by Fannie Mae or Freddie Mac consist of loss share securities, the majority of which, by UPB, are unrated or rated below investment grade at September 30, 2019 by at least one NRSRO. The Company's private label securities are rated investment grade by at least one NRSRO as of September 30, 2019.