Securities Available for Sale and Held to Maturity - Contractual Maturities of Available for Sale and Held to Maturity Securities (Details) - USD ($) $ in Thousands	Mar. 31, 2018	Dec. 31, 2017
Securities Available for Sale, Amortized Cost [Abstract]
Due in one year or less	$ 28,222
Due after one year through five years	54,638
Due after five years through ten years	3,786
Due after ten years	28,993
Total amortized cost, before mortgage-backed securities	115,639
Mortgage-backed securities	2,033,392
Total	2,149,031
Securities Available for Sale, Estimated Fair Value [Abstract]
Due in one year or less	28,280
Due after one year through five years	55,290
Due after five years through ten years	3,745
Due after ten years	28,079
Total fair value, before mortgage-backed securities	115,394
Mortgage-backed securities	1,982,103
Total	2,097,497	$ 2,238,635
Securities Held to Maturity, Amortized Cost [Abstract]
Due in one year or less	155
Due after one year through five years	40,569
Due after five years through ten years	8,990
Due after ten years	0
Total amortized cost, before mortgage-backed securities	49,714
Mortgage-backed securities	974,261
Securities Held to Maturity, Amortized Cost	1,023,975	1,056,486
Securities Held to Maturity, Estimated Fair Value [Abstract]
Due in one year or less	155
Due after one year through five years	41,118
Due after five years through ten years	9,133
Due after ten years	0
Total fair value, before mortgage-backed securities	50,406
Mortgage-backed securities	947,637
Total	$ 998,043	$ 1,046,247