Securities Available for Sale and Held to Maturity - Contractual Maturities of Available for Sale and Held to Maturity Securities (Details) - USD ($) $ in Thousands	Sep. 30, 2016	Dec. 31, 2015
Securities Available for Sale, Amortized Cost [Abstract]
Due in one year or less	$ 32,173
Due after one year through five years	99,840
Due after five years through ten years	8,263
Due after ten years	39,725
Total amortized cost, before mortgage-backed securities	180,001
Mortgage-backed securities	2,192,719
Total	2,372,720
Securities Available for Sale, Estimated Fair Value [Abstract]
Due in one year or less	32,377
Due after one year through five years	103,180
Due after five years through ten years	8,332
Due after ten years	39,269
Total fair value, before mortgage-backed securities	183,158
Mortgage-backed securities	2,227,789
Total	2,410,947	$ 2,345,422
Securities Held to Maturity, Amortized Cost [Abstract]
Due in one year or less	5,845
Due after one year through five years	27,025
Due after five years through ten years	23,703
Due after ten years	0
Total amortized cost, before mortgage-backed securities	56,573
Mortgage-backed securities	1,086,661
Securities Held to Maturity, Amortized Cost	1,143,234	1,187,818
Securities Held to Maturity, Estimated Fair Value [Abstract]
Due in one year or less	5,845
Due after one year through five years	28,099
Due after five years through ten years	25,577
Due after ten years	0
Total fair value, before mortgage-backed securities	59,521
Mortgage-backed securities	1,113,580
Total	$ 1,173,101	$ 1,195,367