Fair Value of Financial Instruments Recorded basis and Fair Value (Details) (USD $) In Thousands, unless otherwise specified	Mar. 31, 2014	Dec. 31, 2013
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Derivative assets	$ 12,064	$ 18,488
Derivative Liabilities, Fair Value Disclosure	11,137	6,681
Carrying (Reported) Amount, Fair Value Disclosure [Member] | Fair Value, Measurements, Recurring [Member]
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities, Fair Value Disclosure	3,959,852	4,018,161
Mortgage loans held for investment, net, Fair Value Disclosure	53,804	55,423
Derivative assets	12,064	18,488
Repurchase Agreements, Fair Value Disclosure	3,485,544	3,580,754
Non-recourse collateralized financing, Fair Value Disclosure	12,394	12,914
Derivative Liabilities, Fair Value Disclosure	11,137	6,681
Estimate of Fair Value, Fair Value Disclosure [Member] | Fair Value, Measurements, Recurring [Member]
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage-backed securities, Fair Value Disclosure	3,959,852	4,018,161
Mortgage loans held for investment, net, Fair Value Disclosure	45,188	46,383
Derivative assets	12,064	18,488
Repurchase Agreements, Fair Value Disclosure	3,485,683	3,580,997
Non-recourse collateralized financing, Fair Value Disclosure	11,915	12,414
Derivative Liabilities, Fair Value Disclosure	$ 11,137	$ 6,681