Investments in RMBS (Details) $ in Thousands		3 Months Ended		9 Months Ended		12 Months Ended
		Sep. 30, 2019 USD ($)	Sep. 30, 2018 USD ($)	Sep. 30, 2019 USD ($) Security	Sep. 30, 2018 USD ($)	Dec. 31, 2018 USD ($) Security
Residential Mortgage-Backed Securities [Abstract]
Carrying value		$ 2,483,237		$ 2,483,237		$ 1,770,110
Carrying value of collateral for repurchase agreements		2,363,218		2,363,218		1,698,688
OTTI charges recognized in earnings		0	$ 0	0	$ 45
RMBS [Member]
Residential Mortgage-Backed Securities [Abstract]
Original face value		2,805,818		2,805,818		2,042,097
Book value		2,448,451		2,448,451		1,808,420
Gross unrealized gains		36,932		36,932		5,606
Gross unrealized losses		(2,146)		(2,146)		(43,916)
Carrying value	[1]	$ 2,483,237		$ 2,483,237		$ 1,770,110
Number of securities | Security				325		247
Weighted average coupon				3.89%		3.98%
Weighted average yield	[2]	3.77%		3.77%		3.82%
Weighted average maturity	[3]			26 years		25 years
Carrying value of collateral for repurchase agreements		$ 2,363,200		$ 2,363,200		$ 1,698,700
RMBS [Member] | Fannie Mae [Member]
Residential Mortgage-Backed Securities [Abstract]
Original face value		1,835,342		1,835,342		1,362,606
Book value		1,598,859		1,598,859		1,208,854
Gross unrealized gains		20,154		20,154		224
Gross unrealized losses		(1,668)		(1,668)		(30,914)
Carrying value	[1]	$ 1,617,345		$ 1,617,345		$ 1,178,164
Number of securities | Security				200		154
Weighted average rating	[4]
Weighted average coupon				3.82%		3.87%
Weighted average yield	[2]	3.67%		3.67%		3.70%
Weighted average maturity	[3]			26 years		25 years
RMBS [Member] | Freddie Mac [Member]
Residential Mortgage-Backed Securities [Abstract]
Original face value		$ 799,247		$ 799,247		$ 548,862
Book value		692,286		692,286		471,148
Gross unrealized gains		10,328		10,328		246
Gross unrealized losses		(478)		(478)		(12,386)
Carrying value	[1]	$ 702,136		$ 702,136		$ 459,008
Number of securities | Security				86		63
Weighted average rating	[4]
Weighted average coupon				3.74%		3.75%
Weighted average yield	[2]	3.60%		3.60%		3.60%
Weighted average maturity	[3]			27 years		27 years
RMBS [Member] | CMOs [Member]
Residential Mortgage-Backed Securities [Abstract]
Original face value		$ 171,229		$ 171,229		$ 130,629
Book value		157,306		157,306		128,418
Gross unrealized gains		6,450		6,450		5,136
Gross unrealized losses		0		0		(616)
Carrying value	[1]	$ 163,756		$ 163,756		$ 132,938
Number of securities | Security				39		30
Weighted average rating	[4]
Weighted average coupon				5.15%		5.79%
Weighted average yield	[2]	5.45%		5.45%		5.78%
Weighted average maturity	[3]			16 years		15 years

[1] See Note 9 regarding the estimation of fair value, which approximates carrying value for all securities.

[2] The weighted average yield is based on the most recent gross monthly interest income, which is then annualized and divided by the book value of settled securities.

[3] The weighted average maturity is based on the timing of expected principal reduction on the assets.

[4] The Company used an implied AAA rating for the Agency RMBS. CMOs issued by Fannie Mae or Freddie Mac consist of loss share securities, the majority of which, by unpaid principal balance ("UPB"), are unrated or rated below investment grade at September 30, 2019 by at least one NRSRO. The Company's private label securities are rated investment grade by at least one NRSRO as of September 30, 2019.