Carrying Values and Fair Values of Assets and Liabilities (Details) - USD ($) $ in Thousands		Sep. 30, 2015	Dec. 31, 2014	Sep. 30, 2014	Dec. 31, 2013
Fair Value Disclosures [Abstract]
MBS, Carrying Value		$ 15,472,580	$ 17,587,010
Mortgage loans held for investment, Carrying Value		265,281	31,460
Mortgage loans held for sale, Carrying Value		9,572
MSR, Carrying Value		170,447
Cash and cash equivalents, Carrying Value		711,148	627,595	$ 441,837	$ 988,705
Unsettled purchased MBS, Carrying Value		65,834	24,792
Receivable for securities sold, Carrying Value		51,729	5,197
Accrued interest receivable, Carrying Value		47,563	54,274
Principal payments receivable, Carrying Value		144,261	111,439
Debt security, held-to-maturity, Carrying Value	[1]	15,000	15,000
Short-term investments, Carrying Value	[1]	30,444	20,252
Interest rate swaps and swaptions, Carrying Value	[2]	1,831	11,050
Forward purchase commitments, Carrying Value	[2]	2,534	16,101
Repurchase agreements, Carrying Value		14,240,809	15,759,831
Warehouse lines of credit, Carrying Value		33,172
FHLB advances, Carrying Value		14,132
Payable for unsettled securities, Carrying Value		66,266	24,750
Accrued interest payable, Carrying Value		2,533	6,968
Interest rate swap liability, Carrying Value	[3]	22,964	42,052
Futures Contracts liability, Carrying Value	[3]	394,221	202,501
Forward purchase commitments, Carrying Value	[3]	1,736	38
MBS, Fair Value		15,472,580	17,587,010
Mortgage loans held for investment, Fair Value		265,281	31,460
Mortgage loans held for sale, Fair Value		9,572
MSR, Fair Value		170,447
Cash and cash equivalents, Fair Value		711,148	627,595
Unsettled purchased MBS, Fair Value		65,834	24,792
Receivable for securities sold, Fair Value		51,729	5,197
Accrued interest receivable, Fair Value		47,563	54,274
Principal payments receivable, Fair Value		144,261	111,439
Debt security, held-to-maturity, Fair Value	[1]	14,620	14,853
Short-term investments, Fair Value	[1]	30,444	20,252
Interest rate swaps and swaptions, Fair Value	[2]	1,831	11,050
Forward purchase commitments, Fair Value	[2]	2,534	16,101
Repurchase agreements, Fair Value		14,240,809	15,759,831
Warehouse lines of credit		33,172
Federal Home Loan Bank advances		14,132
Payable for unsettled securities, Fair Value		66,266	24,750
Accrued interest payable, Fair Value		2,533	6,968
Interest rate swap liability, Fair Value	[3]	22,964	42,052
Futures Contracts liability, Fair Value	[3]	394,221	202,501
Forward purchase commitments, Fair Value	[3]	$ 1,736	$ 38

[1] Included in other investments on the consolidated balance sheets.

[2] Included in derivative assets on the consolidated balance sheets.

[3] Included in derivative liabilities on the consolidated balance sheets.