Residential Mortgage Securities and MSR-Related Assets (Residential and Mortgage Securities) (Details) - USD ($)	6 Months Ended	12 Months Ended
	Jun. 30, 2019	Dec. 31, 2018
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	$ 5,448,848,000	$ 6,646,178,000
Purchase Premiums	91,977,000	112,369,000
Accretable Purchase Discounts	(117,823,000)	(154,942,000)
Discount Designated as Credit Reserve and OTTI	(479,566,000)	(516,116,000)
Amortized Cost	4,944,118,000	6,088,525,000
Gross Unrealized Gains	472,609,000	484,591,000
Gross Unrealized Losses	(23,766,000)	(63,783,000)
Net Unrealized Gain/(Loss)	448,843,000	420,808,000
Fair Value	5,392,961,000	6,509,333,000
Principal payments receivable	682,000	1,000,000.0
Agency MBS
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	2,174,771,000	2,630,630,000
Purchase Premiums	85,061,000	103,008,000
Accretable Purchase Discounts	(23,000)	(24,000)
Discount Designated as Credit Reserve and OTTI	0	0
Amortized Cost	2,260,491,000	2,734,650,000
Gross Unrealized Gains	19,341,000	13,061,000
Gross Unrealized Losses	(22,457,000)	(49,498,000)
Net Unrealized Gain/(Loss)	(3,116,000)	(36,437,000)
Fair Value	2,257,375,000	2,698,213,000
Agency MBS | Fannie Mae
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	1,412,562,000	1,716,340,000
Purchase Premiums	54,319,000	65,930,000
Accretable Purchase Discounts	(23,000)	(24,000)
Discount Designated as Credit Reserve and OTTI	0	0
Amortized Cost	1,466,858,000	1,782,246,000
Gross Unrealized Gains	14,103,000	12,107,000
Gross Unrealized Losses	(17,543,000)	(32,321,000)
Net Unrealized Gain/(Loss)	(3,440,000)	(20,214,000)
Fair Value	1,463,418,000	1,762,032,000
Agency MBS | Freddie Mac
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	757,852,000	909,561,000
Purchase Premiums	30,662,000	36,991,000
Accretable Purchase Discounts	0	0
Discount Designated as Credit Reserve and OTTI	0	0
Amortized Cost	789,196,000	947,588,000
Gross Unrealized Gains	5,188,000	907,000
Gross Unrealized Losses	(4,914,000)	(17,177,000)
Net Unrealized Gain/(Loss)	274,000	(16,270,000)
Fair Value	789,470,000	931,318,000
Agency MBS | Ginnie Mae
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	4,357,000	4,729,000
Purchase Premiums	80,000	87,000
Accretable Purchase Discounts	0	0
Discount Designated as Credit Reserve and OTTI	0	0
Amortized Cost	4,437,000	4,816,000
Gross Unrealized Gains	50,000	47,000
Gross Unrealized Losses	0	0
Net Unrealized Gain/(Loss)	50,000	47,000
Fair Value	4,487,000	4,863,000
Non-Agency MBS
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	2,881,490,000	3,538,804,000
Purchase Premiums	9,000	40,000
Accretable Purchase Discounts	(117,753,000)	(155,025,000)
Discount Designated as Credit Reserve and OTTI	(479,566,000)	(516,116,000)
Amortized Cost	2,284,180,000	2,867,703,000
Gross Unrealized Gains	444,573,000	458,985,000
Gross Unrealized Losses	(483,000)	(8,389,000)
Net Unrealized Gain/(Loss)	444,090,000	450,596,000
Fair Value	2,728,270,000	3,318,299,000
Credit reserve	468,200,000	503,300,000
OTTI	11,300,000	12,800,000
Non-Agency MBS | Expected to Recover Par
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	1,149,835,000	1,536,485,000
Purchase Premiums	9,000	40,000
Accretable Purchase Discounts	(16,996,000)	(21,725,000)
Discount Designated as Credit Reserve and OTTI	0	0
Amortized Cost	1,132,848,000	1,514,800,000
Gross Unrealized Gains	20,933,000	20,520,000
Gross Unrealized Losses	(174,000)	(7,620,000)
Net Unrealized Gain/(Loss)	20,759,000	12,900,000
Fair Value	1,153,607,000	1,527,700,000
Non-Agency MBS | Expected to Recover Less Than Par
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	1,731,655,000	2,002,319,000
Purchase Premiums	0	0
Accretable Purchase Discounts	(100,757,000)	(133,300,000)
Discount Designated as Credit Reserve and OTTI	(479,566,000)	(516,116,000)
Amortized Cost	1,151,332,000	1,352,903,000
Gross Unrealized Gains	423,640,000	438,465,000
Gross Unrealized Losses	(309,000)	(769,000)
Net Unrealized Gain/(Loss)	423,331,000	437,696,000
Fair Value	$ 1,574,663,000	$ 1,790,599,000
Percentage of current face amount of Non-Agency MBS to be recovered	83.00%	85.00%
Total MBS
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	$ 5,056,261,000	$ 6,169,434,000
Purchase Premiums	85,070,000	103,048,000
Accretable Purchase Discounts	(117,776,000)	(155,049,000)
Discount Designated as Credit Reserve and OTTI	(479,566,000)	(516,116,000)
Amortized Cost	4,544,671,000	5,602,353,000
Gross Unrealized Gains	463,914,000	472,046,000
Gross Unrealized Losses	(22,940,000)	(57,887,000)
Net Unrealized Gain/(Loss)	440,974,000	414,159,000
Fair Value	4,985,645,000	6,016,512,000
CRT securities
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	392,587,000	476,744,000
Purchase Premiums	6,907,000	9,321,000
Accretable Purchase Discounts	(47,000)	107,000
Discount Designated as Credit Reserve and OTTI	0	0
Amortized Cost	399,447,000	486,172,000
Gross Unrealized Gains	8,695,000	12,545,000
Gross Unrealized Losses	(826,000)	(5,896,000)
Net Unrealized Gain/(Loss)	7,869,000	6,649,000
Fair Value	407,316,000	492,821,000
Agency MBS, Fair Value Option
Debt Securities, Available-for-sale [Line Items]
Gross Unrealized Gains	4,700,000	0
Gross Unrealized Losses	0	(3,300,000)
Debt Securities, Fair Value Option	616,000,000.0	736,500,000
RPL/NPL MBS
Debt Securities, Available-for-sale [Line Items]
Principal/ Current Face	1,000,000,000.0	1,400,000,000
Amortized Cost	1,000,000,000.0	1,400,000,000
Fair Value	1,000,000,000.0	1,400,000,000
CRT, Fair Value Option
Debt Securities, Available-for-sale [Line Items]
Gross Unrealized Gains	8,400,000	12,500,000
Gross Unrealized Losses	(800,000)	(5,600,000)
Fair Value	$ 380,800,000	$ 477,400,000