Fair Value of Financial Instruments Recorded basis and Fair Value (Details) - USD ($) $ in Thousands		Dec. 31, 2015	Dec. 31, 2014
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Derivative assets		$ 7,835	$ 5,727
Derivative liabilities		41,205	35,898
Recorded Basis [Member]
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Mortgage-backed Securities Available-for-sale, Fair Value Disclosure		3,493,701	3,516,239
Cost Method Investments, Fair Value Disclosure		24,145	39,700
Investment in Federal Home Loan Bank Stock, Fair Value Disclosure		11,475	0
Derivative assets		7,835	5,727
Securities Loaned or Sold under Agreements to Repurchase, Fair Value Disclosure		2,589,420	3,013,110
Federal Home Loan Bank Borrowings, Fair Value Disclosure		520,000	0
Long-term Debt, Fair Value		8,442	10,786
Derivative liabilities		41,205	35,898
Fair Value, Fair Value Disclosure [Member]
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Mortgage-backed Securities Available-for-sale, Fair Value Disclosure		3,493,701	3,516,239
Cost Method Investments, Fair Value Disclosure	[1]	20,849	35,024
Investment in Federal Home Loan Bank Stock, Fair Value Disclosure		11,475	0
Derivative assets		7,835	5,727
Securities Loaned or Sold under Agreements to Repurchase, Fair Value Disclosure	[2]	2,589,420	3,013,110
Federal Home Loan Bank Borrowings, Fair Value Disclosure	[2]	520,000	0
Long-term Debt, Fair Value	[1]	8,102	10,366
Derivative liabilities		$ 41,205	$ 35,898

[1] The Company determines the fair value of its mortgage loans held for investment, net and its non-recourse collateralized financing using internally developed cash flow models with inputs similar to those used to estimate the fair value of the Company's Level 3 non-Agency MBS.

[2] The carrying value of repurchase agreements and FHLB advances generally approximates fair value due to their short term maturities