Repurchase Agreements and Other Advances (Details) - USD ($) $ in Thousands	9 Months Ended	12 Months Ended
	Sep. 30, 2016	Dec. 31, 2015
Assets Sold under Agreements to Repurchase [Line Items]
Unamortized debt issuance expense	$ 207	$ 1,280
Securities Sold under Agreements to Repurchase
Assets Sold under Agreements to Repurchase [Line Items]
Weighted average remaining term-to-interest rate reset of borrowings under repurchase agreements, days	20 days	21 days
Effective repricing period, months	13 months	18 months
Balance of repurchase agreements	$ 8,482,963	$ 7,888,902
Agency MBS | Securities Sold under Agreements to Repurchase
Assets Sold under Agreements to Repurchase [Line Items]
Balance of repurchase agreements	3,356,803	2,727,542
Fair value of securities pledged as collateral under repurchase agreements	$ 3,521,966	$ 2,881,049
Weighted average haircut (percent)	4.61%	4.67%
Legacy Non-Agency MBS | Securities Sold under Agreements to Repurchase
Assets Sold under Agreements to Repurchase [Line Items]
Balance of repurchase agreements	$ 1,786,248	$ 1,960,222
Fair value of securities pledged as collateral under repurchase agreements	$ 2,545,212	$ 2,818,968
Weighted average haircut (percent)	24.18%	25.84%
Non-Agency MBS acquired from consolidated VIEs	$ 186,400	$ 570,500
3 Year Step-up Securities | Securities Sold under Agreements to Repurchase
Assets Sold under Agreements to Repurchase [Line Items]
Balance of repurchase agreements	1,963,760	2,080,163
Fair value of securities pledged as collateral under repurchase agreements	$ 2,523,227	$ 2,625,866
Weighted average haircut (percent)	23.01%	21.05%
U.S. Treasuries | Securities Sold under Agreements to Repurchase
Assets Sold under Agreements to Repurchase [Line Items]
Balance of repurchase agreements	$ 500,930	$ 504,760
Fair value of securities pledged as collateral under repurchase agreements	$ 506,182	$ 507,443
Weighted average haircut (percent)	1.59%	1.60%
CRT securities | Securities Sold under Agreements to Repurchase
Assets Sold under Agreements to Repurchase [Line Items]
Balance of repurchase agreements	$ 241,606	$ 128,465
Fair value of securities pledged as collateral under repurchase agreements	$ 320,852	$ 170,352
Weighted average haircut (percent)	23.53%	25.04%
Residential Whole Loans | Securities Sold under Agreements to Repurchase
Assets Sold under Agreements to Repurchase [Line Items]
Balance of repurchase agreements	$ 633,616	$ 487,750
Fair value of securities pledged as collateral under repurchase agreements	$ 896,946	$ 684,136
Weighted average haircut (percent)	25.69%	27.69%