Use of Special Purpose Entities and Variable Interest Entities (Details) - USD ($)		3 Months Ended		9 Months Ended
		Sep. 30, 2016	Sep. 30, 2015	Sep. 30, 2016	Sep. 30, 2015	Dec. 31, 2015
Variable Interest Entity [Line Items]
Available-for-sale securities		$ 10,277,990,000		$ 10,277,990,000		$ 11,356,643,000
Residential whole loans		1,300,000,000		1,300,000,000		895,100,000
Residential whole loans at carrying value		550,623,000		550,623,000		271,845,000
Residential whole loans, at fair value		$ 797,191,000		$ 797,191,000		623,276,000
Ownership percentage by parent		100.00%		100.00%
Interest income residential whole loans held at carrying value		$ 5,917,000	$ 4,033,000	$ 16,112,000	$ 11,817,000
Net gain on residential whole loans held at fair value		18,701,000	$ 5,565,000	45,052,000	$ 10,823,000
Non-Agency MBS
Variable Interest Entity [Line Items]
Available-for-sale securities		5,908,815,000		5,908,815,000		6,420,817,000
WFMLT Series 2012-RR1
Variable Interest Entity [Line Items]
Securitized debt		0		0		22,100,000
Variable Interest Entity, Primary Beneficiary
Variable Interest Entity [Line Items]
Taxable income from unwind of resecuritization transaction				$ 70,900,000
Taxable income from unwind of resecuritization transaction (in dollars per share)				$ 0.19
Variable Interest Entity, Primary Beneficiary | Non-Agency MBS
Variable Interest Entity [Line Items]
Available-for-sale securities	[1]	$ 187,000,000		$ 187,000,000		$ 598,300,000
Trust, ownership in residential whole loans
Variable Interest Entity [Line Items]
Ownership percentage by parent		100.00%		100.00%

[1] Non-Agency MBS transferred to consolidated VIEs represent assets of the consolidated VIEs that can be used only to settle the obligations of each respective VIE.