Securities Available for Sale and Held to Maturity - Contractual Maturities of Available for Sale and Held to Maturity Securities (Details) - USD ($) $ in Thousands	Dec. 31, 2015	Dec. 31, 2014
Securities Available for Sale, Amortized Cost [Abstract]
Due in one year or less	$ 20,805
Due after one year through five years	113,325
Due after five years through ten years	16,997
Due after ten years	77,167
Total amortized cost, before mortgage-backed securities	228,294
Mortgage-backed securities	2,111,268
Total	2,339,562
Securities Available for Sale, Estimated Fair Value [Abstract]
Due in one year or less	20,982
Due after one year through five years	117,050
Due after five years through ten years	17,562
Due after ten years	76,388
Total fair value, before mortgage-backed securities	231,982
Mortgage-backed securities	2,113,440
Total	2,345,422	$ 2,374,567
Securities Held to Maturity, Amortized Cost [Abstract]
Due in one year or less	6,300
Due after one year through five years	60,315
Due after five years through ten years	91,059
Due after ten years	0
Total amortized cost, before mortgage-backed securities	157,674
Mortgage-backed securities	1,030,144
Amortized Cost	1,187,818	1,170,685
Securities Held to Maturity, Estimated Fair Value [Abstract]
Due in one year or less	6,505
Due after one year through five years	61,936
Due after five years through ten years	95,433
Due after ten years	0
Total fair value, before mortgage-backed securities	163,874
Mortgage-backed securities	1,031,493
Total	$ 1,195,367	$ 1,182,846