Repurchase Agreements (Details) - USD ($) $ in Thousands	6 Months Ended	12 Months Ended
	Jun. 30, 2015	Dec. 31, 2014
Assets Sold under Agreements to Repurchase [Line Items]
Balance of repurchase agreements	$ 9,635,036	$ 8,267,388
Securities Sold under Agreements to Repurchase
Assets Sold under Agreements to Repurchase [Line Items]
Weighted average remaining term-to-interest rate reset of borrowings under repurchase agreements	23 days	25 days
Effective repricing period	16 months	21 months
Balance of repurchase agreements	$ 9,635,036	$ 8,267,388
Agency Mortgage Backed Securities | Securities Sold under Agreements to Repurchase
Assets Sold under Agreements to Repurchase [Line Items]
Balance of repurchase agreements	4,672,707	5,177,835
Fair value of securities pledged as collateral under repurchase agreements	$ 4,923,712	$ 5,462,566
Weighted average haircut (percent)	4.65%	4.79%
Legacy Non-Agency MBS | Securities Sold under Agreements to Repurchase
Assets Sold under Agreements to Repurchase [Line Items]
Balance of repurchase agreements	$ 2,201,596	$ 2,233,236
Fair value of securities pledged as collateral under repurchase agreements	$ 3,139,828	$ 3,491,312
Weighted average haircut (percent)	25.57%	28.88%
Non-Agency MBS acquired from consolidated VIEs	$ 587,100	$ 1,275,000
RPL/NPL MBS | Securities Sold under Agreements to Repurchase
Assets Sold under Agreements to Repurchase [Line Items]
Balance of repurchase agreements	2,044,697	130,919
Fair value of securities pledged as collateral under repurchase agreements	$ 2,591,503	$ 160,688
Weighted average haircut (percent)	21.81%	20.00%
U.S. Treasuries | Securities Sold under Agreements to Repurchase
Assets Sold under Agreements to Repurchase [Line Items]
Balance of repurchase agreements	$ 491,215	$ 507,114
Fair value of securities pledged as collateral under repurchase agreements	$ 498,336	$ 512,105
Weighted average haircut (percent)	1.65%	1.62%
CRT securities | Securities Sold under Agreements to Repurchase
Assets Sold under Agreements to Repurchase [Line Items]
Balance of repurchase agreements	$ 92,921	$ 75,960
Fair value of securities pledged as collateral under repurchase agreements	$ 116,690	$ 94,610
Weighted average haircut (percent)	25.18%	25.00%
Residential whole loans | Securities Sold under Agreements to Repurchase
Assets Sold under Agreements to Repurchase [Line Items]
Balance of repurchase agreements	$ 131,900	$ 142,324
Fair value of securities pledged as collateral under repurchase agreements	$ 201,062	$ 212,986
Weighted average haircut (percent)	33.74%	33.43%