Investments in RMBS (Details) $ in Thousands		12 Months Ended
		Dec. 31, 2019 USD ($) Security	Dec. 31, 2018 USD ($) Security
Residential Mortgage-Backed Securities [Abstract]
Carrying value		$ 2,508,360	$ 1,770,110
Carrying value of collateral for repurchase agreements		2,419,539	1,698,688
OTTI charges recognized in earnings		0	45
RMBS [Member]
Residential Mortgage-Backed Securities [Abstract]
Original face value		2,880,949	2,042,097
Book value		2,466,828	1,808,420
Gross unrealized gains		42,468	5,606
Gross unrealized losses		(936)	(43,916)
Carrying value	[1]	$ 2,508,360	$ 1,770,110
Number of securities | Security		327	247
Weighted average coupon		3.85%	3.98%
Weighted average yield	[2]	3.72%	3.82%
Weighted average maturity	[3]	26 years	25 years
Carrying value of collateral for repurchase agreements		$ 2,419,500	$ 1,698,700
RMBS [Member] | Fannie Mae [Member]
Residential Mortgage-Backed Securities [Abstract]
Original face value		1,878,229	1,362,606
Book value		1,596,288	1,208,854
Gross unrealized gains		23,636	224
Gross unrealized losses		(691)	(30,914)
Carrying value	[1]	$ 1,619,233	$ 1,178,164
Number of securities | Security		198	154
Weighted average rating	[4]
Weighted average coupon		3.80%	3.87%
Weighted average yield	[2]	3.65%	3.70%
Weighted average maturity	[3]	27 years	25 years
RMBS [Member] | Freddie Mac [Member]
Residential Mortgage-Backed Securities [Abstract]
Original face value		$ 824,991	$ 548,862
Book value		715,892	471,148
Gross unrealized gains		12,204	246
Gross unrealized losses		(245)	(12,386)
Carrying value	[1]	$ 727,851	$ 459,008
Number of securities | Security		88	63
Weighted average rating	[4]
Weighted average coupon		3.72%	3.75%
Weighted average yield	[2]	3.59%	3.60%
Weighted average maturity	[3]	28 years	27 years
RMBS [Member] | CMOs [Member]
Residential Mortgage-Backed Securities [Abstract]
Original face value		$ 127,229	$ 100,129
Book value		123,053	99,023
Gross unrealized gains		6,030	5,060
Gross unrealized losses		0	(583)
Carrying value	[1]	$ 129,083	$ 103,500
Number of securities | Security		30	23
Weighted average rating	[4]
Weighted average coupon		5.28%	6.30%
Weighted average yield	[2]	5.26%	6.29%
Weighted average maturity	[3]	11 years	11 years
RMBS [Member] | Private Label MBS [Member]
Residential Mortgage-Backed Securities [Abstract]
Original face value		$ 50,500	$ 30,500
Book value		31,595	29,395
Gross unrealized gains		598	76
Gross unrealized losses		0	(33)
Carrying value	[1]	$ 32,193	$ 29,438
Number of securities | Security		11	7
Weighted average rating	[4]
Weighted average coupon		4.06%	4.10%
Weighted average yield	[2]	4.06%	4.11%
Weighted average maturity	[3]	29 years	29 years

[1] See Note 9 regarding the estimation of fair value, which approximates carrying value for all securities.

[2] The weighted average yield is based on the most recent gross monthly interest income, which is then annualized and divided by the book value of settled securities.

[3] The weighted average maturity is based on the timing of expected principal reduction on the assets.

[4] The Company used an implied AAA rating for the Agency RMBS. Collateralized mortgage obligations ("CMOs") issued by Fannie Mae or Freddie Mac consist of loss share securities, approximately half of which, by unpaid principal balance ("UPB"), are unrated or rated below investment grade at December 31, 2019 by at least one NRSRO. Private label securities are rated investment grade or better by at least one NRSRO as of December 31, 2019.