Securities Available for Sale and Held to Maturity - Contractual Maturities of Available for Sale and Held to Maturity Securities (Details) - USD ($) $ in Thousands	Sep. 30, 2018	Dec. 31, 2017
Securities Available for Sale, Amortized Cost [Abstract]
Due in one year or less	$ 25,525
Due after one year through five years	37,426
Due after five years through ten years	3,124
Due after ten years	24,563
Total amortized cost, before mortgage-backed securities	90,638
Mortgage-backed securities	1,844,464
Total	1,935,102
Securities Available for Sale, Estimated Fair Value [Abstract]
Due in one year or less	25,554
Due after one year through five years	37,661
Due after five years through ten years	3,054
Due after ten years	23,366
Total fair value, before mortgage-backed securities	89,635
Mortgage-backed securities	1,774,998
Total	1,864,633	$ 2,238,635
Securities Held to Maturity, Amortized Cost [Abstract]
Due in one year or less	170
Due after one year through five years	41,912
Due after five years through ten years	4,266
Due after ten years	0
Total amortized cost, before mortgage-backed securities	46,348
Mortgage-backed securities	897,535
Securities Held to Maturity, Amortized Cost	943,883	1,056,486
Securities Held to Maturity, Estimated Fair Value [Abstract]
Due in one year or less	170
Due after one year through five years	41,934
Due after five years through ten years	4,287
Due after ten years	0
Total fair value, before mortgage-backed securities	46,391
Mortgage-backed securities	860,926
Total	$ 907,317	$ 1,046,247