Residential Mortgage Securities and MSR-Related Assets (Unrealized Losses) (Details) $ in Thousands	Mar. 31, 2019 USD ($) security	Dec. 31, 2018 USD ($)
Fair Value
Less than 12 Months	$ 541,010
12 Months or more	1,230,636
Total	1,771,646
Unrealized Losses
Less than 12 Months	3,597
12 Months or more	35,681
Total	$ 39,278
Number of Securities
Less than 12 months (in security) | security	99
12 months or more (in security) | security	434
Available-for-sale securities	$ 6,069,571	$ 6,509,333
Gross Unrealized Losses	39,278	63,783
Agency MBS
Fair Value
Less than 12 Months	100,197
12 Months or more	1,137,370
Total	1,237,567
Unrealized Losses
Less than 12 Months	859
12 Months or more	34,653
Total	$ 35,512
Number of Securities
Less than 12 months (in security) | security	53
12 months or more (in security) | security	429
Available-for-sale securities	$ 2,546,597	2,698,213
Gross Unrealized Losses	35,512	49,498
Agency MBS | Fannie Mae
Fair Value
Less than 12 Months	99,955
12 Months or more	860,036
Total	959,991
Unrealized Losses
Less than 12 Months	858
12 Months or more	25,023
Total	$ 25,881
Number of Securities
Less than 12 months (in security) | security	52
12 months or more (in security) | security	303
Available-for-sale securities	$ 1,641,215	1,762,032
Gross Unrealized Losses	25,881	32,321
Agency MBS | Freddie Mac
Fair Value
Less than 12 Months	242
12 Months or more	277,334
Total	277,576
Unrealized Losses
Less than 12 Months	1
12 Months or more	9,630
Total	$ 9,631
Number of Securities
Less than 12 months (in security) | security	1
12 months or more (in security) | security	126
Available-for-sale securities	$ 900,691	931,318
Gross Unrealized Losses	9,631	17,177
Non-Agency MBS
Fair Value
Less than 12 Months	304,053
12 Months or more	93,266
Total	397,319
Unrealized Losses
Less than 12 Months	952
12 Months or more	1,028
Total	$ 1,980
Number of Securities
Less than 12 months (in security) | security	13
12 months or more (in security) | security	5
Available-for-sale securities	$ 3,099,272	3,318,299
Gross Unrealized Losses	1,980	8,389
Non-Agency MBS | Expected to Recover Par
Fair Value
Less than 12 Months	290,671
12 Months or more	91,257
Total	381,928
Unrealized Losses
Less than 12 Months	636
12 Months or more	966
Total	$ 1,602
Number of Securities
Less than 12 months (in security) | security	5
12 months or more (in security) | security	4
Available-for-sale securities	$ 1,424,260	1,527,700
Gross Unrealized Losses	1,602	7,620
Non-Agency MBS | Expected to Recover Less Than Par
Fair Value
Less than 12 Months	13,382
12 Months or more	2,009
Total	15,391
Unrealized Losses
Less than 12 Months	316
12 Months or more	62
Total	$ 378
Number of Securities
Less than 12 months (in security) | security	8
12 months or more (in security) | security	1
Available-for-sale securities	$ 1,675,012	1,790,599
Gross Unrealized Losses	378	769
Total MBS
Fair Value
Less than 12 Months	404,250
12 Months or more	1,230,636
Total	1,634,886
Unrealized Losses
Less than 12 Months	1,811
12 Months or more	35,681
Total	$ 37,492
Number of Securities
Less than 12 months (in security) | security	66
12 months or more (in security) | security	434
Available-for-sale securities	$ 5,645,869	6,016,512
Gross Unrealized Losses	37,492	57,887
CRT securities
Fair Value
Less than 12 Months	136,760
12 Months or more	0
Total	136,760
Unrealized Losses
Less than 12 Months	1,786
12 Months or more	0
Total	$ 1,786
Number of Securities
Less than 12 months (in security) | security	33
12 months or more (in security) | security	0
Available-for-sale securities	$ 423,702	492,821
Gross Unrealized Losses	1,786	$ 5,896
CRT in Loss Positions Only, Fair Value Option
Number of Securities
Available-for-sale securities	136,800
Gross Unrealized Losses	$ 1,800