Fair Value of Financial Instruments Recorded basis and Fair Value (Details) - USD ($) $ in Thousands		Dec. 31, 2016	Dec. 31, 2015
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Derivative assets		$ 28,534	$ 7,835
Derivative liabilities		6,922	41,205
Carrying Value
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Mortgage-backed Securities Available-for-sale, Fair Value Disclosure		3,212,084	3,493,701
Mortgage loans held for investment, net, Fair Value Disclosure		19,036	24,145
Investment in FHLB stock, Fair Value Disclosure		9	11,475
Derivative assets		28,534	7,835
Repurchase agreements, Fair Value Disclosure		2,898,952	2,589,420
FHLB advances, Fair Value Disclosure		0	520,000
Non-recourse collateralized financing, Fair Value		6,440	8,442
Derivative liabilities		6,922	41,205
Fair Value
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Mortgage-backed Securities Available-for-sale, Fair Value Disclosure		3,212,084	3,493,701
Mortgage loans held for investment, net, Fair Value Disclosure	[1]	15,971	20,849
Investment in FHLB stock, Fair Value Disclosure	[2]	9	11,475
Derivative assets		28,534	7,835
Repurchase agreements, Fair Value Disclosure	[2]	2,898,952	2,589,420
FHLB advances, Fair Value Disclosure	[2]	0	520,000
Non-recourse collateralized financing, Fair Value	[1]	6,357	8,102
Derivative liabilities		$ 6,922	$ 41,205

[1] The Company determines the fair value of its mortgage loans held for investment, net and its non-recourse collateralized financing using internally developed cash flow models with inputs similar to those used to estimate the fair value of the Company's Level 3 non-Agency MBS.

[2] The carrying value of repurchase agreements and FHLB advances generally approximates fair value due to their short term maturities