Fair Value of Financial Instruments Recorded basis and Fair Value (Details) (USD $) In Thousands, unless otherwise specified	Sep. 30, 2014		Dec. 31, 2013
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative assets	$ 7,297		$ 18,488
Derivative Liabilities, Fair Value Disclosure	18,058		6,681
Carrying (Reported) Amount, Fair Value Disclosure [Member] | Fair Value, Measurements, Recurring [Member]
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities, Fair Value Disclosure	3,615,393		4,018,161
Mortgage loans held for investment, net, Fair Value Disclosure	41,454		55,423
Derivative assets	7,297		18,488
Repurchase Agreements, Fair Value Disclosure	3,150,254		3,580,754
Non-recourse collateralized financing, Fair Value Disclosure	11,194		12,914
Derivative Liabilities, Fair Value Disclosure	18,058		6,681	[1]
Estimate of Fair Value, Fair Value Disclosure [Member] | Fair Value, Measurements, Recurring [Member]
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities, Fair Value Disclosure	3,615,393		4,018,161
Mortgage loans held for investment, net, Fair Value Disclosure	36,590	[2]	46,383	[2]
Derivative assets	7,297		18,488
Repurchase Agreements, Fair Value Disclosure	3,150,254		3,580,997	[1]
Non-recourse collateralized financing, Fair Value Disclosure	10,757	[2]	12,414	[2]
Derivative Liabilities, Fair Value Disclosure	$ 18,058		$ 6,681	[1]

[1] (2) The difference between the recorded basis of repurchase agreements and their fair value as of December 31, 2013 is the unamortized deferred costs remaining for the 2-year repurchase facility.

[2] (1) The Company determines the fair value of its mortgage loans held for investment, net and its non-recourse collateralized financing using internally developed cash flow models with inputs similar to those used to estimate fair value of the Company's Level 3 non-Agency MBS.