Fair Value of Financial Instruments - Assets and Liabilities on Recurring Basis (Details) - USD ($)		Dec. 31, 2015		Dec. 31, 2014
Assets:
MBS and CRT securities		$ 11,356,643,000		$ 10,762,622,000
Securities obtained and pledged as collateral		507,443,000		512,105,000
Residential whole loans, at fair value		623,276,000		143,472,000
Linked Transactions		0		398,336,000
Swaps		1,127,000		3,136,000
Liabilities:
Obligation to return securities obtained as collateral		507,443,000		512,105,000
Agency MBS
Assets:
MBS and CRT securities		4,752,244,000		5,904,207,000
Non-Agency MBS
Assets:
MBS and CRT securities	[1]	6,420,817,000		4,755,432,000
CRT securities
Assets:
MBS and CRT securities		183,582,000	[2]	102,983,000
Level 2
Components of financial instruments carried at fair value
Derivative credit risk valuation adjustment		0
Recurring basis
Assets:
Securities obtained and pledged as collateral		507,443,000		512,105,000
Linked Transactions				398,336,000
Swaps		1,127,000		3,136,000
Assets, Fair Value Disclosure		12,488,489,000		11,819,671,000
Liabilities:
Swaps		70,526,000		62,198,000
Obligation to return securities obtained as collateral		507,443,000		512,105,000
Financial and Nonfinancial Liabilities, Fair Value Disclosure		577,969,000		574,303,000
Recurring basis | Agency MBS
Assets:
MBS and CRT securities		4,752,244,000		5,904,207,000
Recurring basis | Non-Agency MBS
Assets:
MBS and CRT securities		6,420,817,000		4,755,432,000
Recurring basis | CRT securities
Assets:
MBS and CRT securities		183,582,000		102,983,000
Recurring basis | Residential whole loans
Assets:
Residential whole loans, at fair value		623,276,000		143,472,000
Recurring basis | Level 1
Assets:
Securities obtained and pledged as collateral		507,443,000		512,105,000
Linked Transactions				0
Swaps		0		0
Assets, Fair Value Disclosure		507,443,000		512,105,000
Liabilities:
Swaps		0		0
Obligation to return securities obtained as collateral		507,443,000		512,105,000
Financial and Nonfinancial Liabilities, Fair Value Disclosure		507,443,000		512,105,000
Recurring basis | Level 1 | Agency MBS
Assets:
MBS and CRT securities		0		0
Recurring basis | Level 1 | Non-Agency MBS
Assets:
MBS and CRT securities		0		0
Recurring basis | Level 1 | CRT securities
Assets:
MBS and CRT securities		0		0
Recurring basis | Level 1 | Residential whole loans
Assets:
Residential whole loans, at fair value		0		0
Recurring basis | Level 2
Assets:
Securities obtained and pledged as collateral		0		0
Linked Transactions				398,336,000
Swaps		1,127,000		3,136,000
Assets, Fair Value Disclosure		11,357,770,000		11,164,094,000
Liabilities:
Swaps		70,526,000		62,198,000
Obligation to return securities obtained as collateral		0		0
Financial and Nonfinancial Liabilities, Fair Value Disclosure		70,526,000		62,198,000
Recurring basis | Level 2 | Agency MBS
Assets:
MBS and CRT securities		4,752,244,000		5,904,207,000
Recurring basis | Level 2 | Non-Agency MBS
Assets:
MBS and CRT securities		6,420,817,000		4,755,432,000
Recurring basis | Level 2 | CRT securities
Assets:
MBS and CRT securities		183,582,000		102,983,000
Recurring basis | Level 2 | Residential whole loans
Assets:
Residential whole loans, at fair value		0		0
Recurring basis | Level 3
Assets:
Securities obtained and pledged as collateral		0		0
Linked Transactions				0
Swaps		0		0
Assets, Fair Value Disclosure		623,276,000		143,472,000
Liabilities:
Swaps		0		0
Obligation to return securities obtained as collateral		0		0
Financial and Nonfinancial Liabilities, Fair Value Disclosure		0		0
Recurring basis | Level 3 | Agency MBS
Assets:
MBS and CRT securities		0		0
Recurring basis | Level 3 | Non-Agency MBS
Assets:
MBS and CRT securities		0		0
Recurring basis | Level 3 | CRT securities
Assets:
MBS and CRT securities		0		0
Recurring basis | Level 3 | Residential whole loans
Assets:
Residential whole loans, at fair value		$ 623,276,000		$ 143,472,000

[1] At December 31, 2015 and 2014, the Company expected to recover approximately 89% and 83%, respectively, of the then-current face amount of Non-Agency MBS.

[2] Amounts disclosed at December 31, 2015 includes CRT securities with a fair value of $62.2 million for which the fair value option has been elected. Such securities have gross unrealized gains of approximately $332,000, gross unrealized losses of approximately $555,000 and net unrealized losses of approximately $223,000 at December 31, 2015.