Investments in RMBS (Details) $ in Thousands		12 Months Ended
		Dec. 31, 2018 USD ($) Security	Dec. 31, 2017 USD ($) Security
Residential Mortgage-Backed Securities [Abstract]
Carrying value		$ 1,770,110	$ 1,840,912
Carrying value of collateral for repurchase agreements		1,698,688	1,728,564
OTTI charges recognized in earnings		45	77
RMBS [Member]
Residential Mortgage-Backed Securities [Abstract]
Original face value		2,042,097	1,961,352
Book value		1,808,420	1,843,855
Gross unrealized gains		5,606	8,634
Gross unrealized losses		(43,916)	(11,577)
Carrying value	[1]	$ 1,770,110	$ 1,840,912
Number of securities | Security		247	238
Weighted average coupon		3.98%	3.86%
Weighted average yield	[2]	3.82%	3.66%
Weighted average maturity	[3]	25 years	25 years
Carrying value of collateral for repurchase agreements		$ 1,698,700	$ 1,728,600
RMBS [Member] | Fannie Mae [Member]
Residential Mortgage-Backed Securities [Abstract]
Original face value		1,362,606	1,306,823
Book value		1,208,854	1,241,027
Gross unrealized gains		224	1,427
Gross unrealized losses		(30,914)	(8,755)
Carrying value	[1]	$ 1,178,164	$ 1,233,699
Number of securities | Security		154	154
Weighted average rating		(B)
Weighted average coupon		3.87%	3.80%
Weighted average yield	[2]	3.70%	3.61%
Weighted average maturity	[3]	25 years	26 years
RMBS [Member] | Freddie Mac [Member]
Residential Mortgage-Backed Securities [Abstract]
Original face value		$ 548,862	$ 556,204
Book value		471,148	515,475
Gross unrealized gains		246	864
Gross unrealized losses		(12,386)	(2,795)
Carrying value	[1]	$ 459,008	$ 513,544
Number of securities | Security		63	64
Weighted average rating		(B)
Weighted average coupon		3.75%	3.74%
Weighted average yield	[2]	3.60%	3.57%
Weighted average maturity	[3]	27 years	27 years
RMBS [Member] | CMOs [Member]
Residential Mortgage-Backed Securities [Abstract]
Original face value		$ 130,629	$ 98,325
Book value		128,418	87,353
Gross unrealized gains		5,136	6,343
Gross unrealized losses		(616)	(27)
Carrying value	[1]	$ 132,938	$ 93,669
Number of securities | Security		30	20
Weighted average rating		(B)
Weighted average coupon		5.79%	5.26%
Weighted average yield	[2]	5.78%	4.88%
Weighted average maturity	[3]	15 years	12 years

[1] See Note 9 regarding the estimation of fair value, which approximates carrying value for all securities.

[2] The weighted average yield is based on the most recent gross monthly interest income, which is then annualized and divided by the book value of settled securities.

[3] The weighted average maturity is based on the timing of expected principal reduction on the assets.