Securities Available for Sale and Held to Maturity - Contractual Maturities of Available for Sale and Held to Maturity Securities (Details) - USD ($) $ in Thousands	Dec. 31, 2017	Dec. 31, 2016
Securities Available for Sale, Amortized Cost [Abstract]
Due in one year or less	$ 31,376
Due after one year through five years	57,372
Due after five years through ten years	3,866
Due after ten years	31,582
Total amortized cost, before mortgage-backed securities	124,196
Mortgage-backed securities	2,137,934
Total	2,262,130
Securities Available for Sale, Estimated Fair Value [Abstract]
Due in one year or less	31,525
Due after one year through five years	58,343
Due after five years through ten years	3,845
Due after ten years	30,801
Total fair value, before mortgage-backed securities	124,514
Mortgage-backed securities	2,114,121
Total	2,238,635	$ 2,356,682
Securities Held to Maturity, Amortized Cost [Abstract]
Due in one year or less	155
Due after one year through five years	40,594
Due after five years through ten years	8,982
Due after ten years	0
Total amortized cost, before mortgage-backed securities	49,731
Mortgage-backed securities	1,006,755
Securities Held to Maturity, Amortized Cost	1,056,486	1,158,643
Securities Held to Maturity, Estimated Fair Value [Abstract]
Due in one year or less	155
Due after one year through five years	41,450
Due after five years through ten years	9,293
Due after ten years	0
Total fair value, before mortgage-backed securities	50,898
Mortgage-backed securities	995,349
Total	$ 1,046,247	$ 1,157,046