Investments in RMBS, Assets by Maturity (Details) $ in Thousands		12 Months Ended
		Dec. 31, 2019 USD ($) Security	Dec. 31, 2018 USD ($) Security
RMBS, Assets by Maturity [Abstract]
Carrying value		$ 2,508,360	$ 1,770,110
RMBS [Member]
RMBS, Assets by Maturity [Abstract]
Original face value		2,880,949	2,042,097
Book value		2,466,828	1,808,420
Gross unrealized gains		42,468	5,606
Gross unrealized losses		(936)	(43,916)
Carrying value	[1]	$ 2,508,360	$ 1,770,110
Number of securities | Security		327	247
Weighted average coupon		3.85%	3.98%
Weighted average yield	[2]	3.72%	3.82%
Weighted average maturity	[3]	26 years	25 years
RMBS [Member] | 1-5 Years [Member]
RMBS, Assets by Maturity [Abstract]
Original face value		$ 1,500
Book value		895
Gross unrealized gains		64
Gross unrealized losses		0
Carrying value	[1]	$ 959
Number of securities | Security		1
Weighted average rating	[4]
Weighted average coupon		6.34%
Weighted average yield	[2]	6.34%
Weighted average maturity	[3]	4 years
RMBS [Member] | 5-10 Years [Member]
RMBS, Assets by Maturity [Abstract]
Original face value		$ 64,579	$ 24,377
Book value		61,935	15,100
Gross unrealized gains		4,153	731
Gross unrealized losses		0	(134)
Carrying value	[1]	$ 66,088	$ 15,697
Number of securities | Security		13	7
Weighted average rating	[4]
Weighted average coupon		5.85%	4.97%
Weighted average yield	[2]	5.81%	4.93%
Weighted average maturity	[3]	9 years	9 years
RMBS [Member] | Over 10 Years [Member]
RMBS, Assets by Maturity [Abstract]
Original face value		$ 2,814,870	$ 2,017,720
Book value		2,403,998	1,793,320
Gross unrealized gains		38,251	4,875
Gross unrealized losses		(936)	(43,782)
Carrying value	[1]	$ 2,441,313	$ 1,754,413
Number of securities | Security		313	240
Weighted average rating	[4]
Weighted average coupon		3.80%	3.97%
Weighted average yield	[2]	3.66%	3.81%
Weighted average maturity	[3]	27 years	25 years

[1] See Note 9 regarding the estimation of fair value, which approximates carrying value for all securities.

[2] The weighted average yield is based on the most recent gross monthly interest income, which is then annualized and divided by the book value of settled securities.

[3] The weighted average maturity is based on the timing of expected principal reduction on the assets.

[4] The Company used an implied AAA rating for the Agency RMBS. CMOs issued by Fannie Mae or Freddie Mac consist of loss share securities, approximately half of which, by UPB, are unrated or rated below investment grade at December 31, 2019 by at least one NRSRO. Private label securities are rated investment grade or better by at least one NRSRO as of December 31, 2019.