Income Taxes - Deferred Tax Asset and Liabilities (Details) (USD $) In Thousands, unless otherwise specified	Dec. 31, 2013	Dec. 31, 2012
Income Tax Disclosure [Abstract]
Deferred Tax Assets, Basis differences related to financial derivatives	$ 22,349	$ 44,963
Deferred Tax Asset - Basis differences related to securities	2,509	0
Deferred Tax Assets, Unrealized Losses on Available-for-Sale Securities	8,762	0
Deferred Tax Assets, Allowance for losses	4,667	5,911
Deferred Tax Assets, Stock-based compensation	1,916	2,111
Deferred Tax Assets, Capital Loss Carryforwards	38,532	39,272
Capital Loss Carryforwards, Valuation Allowance	(36,432)	(39,272)
Deferred tax asset lower of cost or fair value on loans held for sale	0	2,080
Deferred tax asset, Amortization of premiums on capital investments	1,499	1,352
Valuation Allowance, amortization of premiums on capital investments	(1,499)	(1,352)
Deferred Tax Assets, Other	2,455	2,333
Deferred Tax Assets, Gross	44,758	57,398
Deferred tax liability, Basis difference related to securities	0	13,160
Deferred tax liabilities, Unrealized gains on available-for-sale securities	0	39,829
Deferred tax liability, Basis difference in subsidiary	353	1,118
Deferred Tax Liabilities, Other	360	168
Deferred Tax Liabilities, Gross	713	54,275
Deferred Tax Asset Liability Net	$ 44,045	$ 3,123