Carrying Values and Fair Values of Assets and Liabilities (Details) - USD ($) $ in Thousands		Dec. 31, 2015	Dec. 31, 2014	Dec. 31, 2013	Dec. 31, 2012
Fair Value Disclosures [Abstract]
MBS, Carrying Value		$ 14,302,230	$ 17,587,010
Agency CRT securities, Carrying Value		109,387
Mortgage loans held for investment, Carrying Value		116,857	31,460
Mortgage loans held for investment in securitization trusts, Carrying Value		226,908
Mortgage loans held for sale, Carrying Value		17,542
MSR, Carrying Value		269,926
Cash and cash equivalents, Carrying Value		816,715	627,595	$ 988,705	$ 449,445
Unsettled purchased MBS, Carrying Value		12,582	24,792
Receivable for securities sold, Carrying Value			5,197
Accrued interest receivable, Carrying Value		45,008	54,274
Principal payments receivable, Carrying Value		108,201	111,439
Debt security, held-to-maturity, Carrying Value	[1]	15,000	15,000
Short-term investments, Carrying Value	[1]	30,327	20,252
Interest rate swaps and swaptions, Carrying Value	[2]	2,031	11,050
Futures Contracts asset, Carrying Value	[2]	693
Forward purchase commitments, Carrying Value	[2]	190	16,101
Repurchase agreements, Carrying Value		13,443,883	15,759,831
Warehouse lines of credit, Carrying Value		60,096
FHLB advances, Carrying Value		14,132
Collateralized borrowings in securitization trusts, Carrying Value		57,611
Payable for unsettled securities, Carrying Value		12,582	24,750
Accrued interest payable, Carrying Value		4,938	6,968
Interest rate swap liability, Carrying Value	[3]	6,802	42,052
Futures Contracts liability, Carrying Value	[3]	286,058	202,501
Forward purchase commitments, Carrying Value	[3]	32,373	38
MBS, Fair Value		14,302,230	17,587,010
Agency CRT securities, at fair value		109,387
Mortgage loans held for investment, Fair Value		116,857	31,460
Mortgage loans held for investment in securitization trusts, at fair value		226,908
Mortgage loans held for sale, Fair Value		17,542
MSR, Fair Value		269,926
Cash and cash equivalents, Fair Value		816,715	627,595
Unsettled purchased MBS, Fair Value		12,582	24,792
Receivable for securities sold, Fair Value			5,197
Accrued interest receivable, Fair Value		45,008	54,274
Principal payments receivable, Fair Value		108,201	111,439
Debt security, held-to-maturity, Fair Value	[1]	14,871	14,853
Short-term investments, Fair Value	[1]	30,327	20,252
Interest rate swaps and swaptions, Fair Value	[2]	2,031	11,050
Futures Contracts asset, Fair Value	[2]	693
Forward purchase commitments, Fair Value	[2]	190	16,101
Repurchase agreements, Fair Value		13,443,883	15,759,831
Warehouse lines of credit		60,096
FHLB advances, Fair Value		14,132
Collateralized borrowings in securitization trusts		57,611
Payable for unsettled securities, Fair Value		12,582	24,750
Accrued interest payable, Fair Value		4,938	6,968
Interest rate swap liability, Fair Value	[3]	6,802	42,052
Futures Contracts liability, Fair Value	[3]	286,058	202,501
Forward purchase commitments, Fair Value	[3]	$ 32,373	$ 38

[1] Included in other investments on the consolidated balance sheets.

[2] Included in derivative assets on the consolidated balance sheets.

[3] Included in derivative liabilities on the consolidated balance sheets.