Securities Available for Sale and Held to Maturity - Contractual Maturities of Available for Sale and Held to Maturity Securities (Details) - USD ($) $ in Thousands	Dec. 31, 2016	Dec. 31, 2015
Securities Available for Sale, Amortized Cost [Abstract]
Due in one year or less	$ 33,592
Due after one year through five years	90,084
Due after five years through ten years	7,500
Due after ten years	38,894
Total amortized cost, before mortgage-backed securities	170,070
Mortgage-backed securities	2,196,098
Total	2,366,168
Securities Available for Sale, Estimated Fair Value [Abstract]
Due in one year or less	33,793
Due after one year through five years	92,100
Due after five years through ten years	7,522
Due after ten years	37,997
Total fair value, before mortgage-backed securities	171,412
Mortgage-backed securities	2,185,270
Total	2,356,682	$ 2,345,422
Securities Held to Maturity, Amortized Cost [Abstract]
Due in one year or less	145
Due after one year through five years	35,983
Due after five years through ten years	13,822
Due after ten years	0
Total amortized cost, before mortgage-backed securities	49,950
Mortgage-backed securities	1,108,693
Securities Held to Maturity, Amortized Cost	1,158,643	1,187,818
Securities Held to Maturity, Estimated Fair Value [Abstract]
Due in one year or less	145
Due after one year through five years	37,249
Due after five years through ten years	14,360
Due after ten years	0
Total fair value, before mortgage-backed securities	51,754
Mortgage-backed securities	1,105,292
Total	$ 1,157,046	$ 1,195,367