Securities Available for Sale and Held to Maturity - Contractual Maturities of Available for Sale and Held to Maturity Securities (Details) - USD ($) $ in Thousands	Jun. 30, 2018	Dec. 31, 2017
Securities Available for Sale, Amortized Cost [Abstract]
Due in one year or less	$ 29,282
Due after one year through five years	43,102
Due after five years through ten years	3,392
Due after ten years	26,594
Total amortized cost, before mortgage-backed securities	102,370
Mortgage-backed securities	1,933,114
Total	2,035,484
Securities Available for Sale, Estimated Fair Value [Abstract]
Due in one year or less	29,393
Due after one year through five years	43,515
Due after five years through ten years	3,331
Due after ten years	25,523
Total fair value, before mortgage-backed securities	101,762
Mortgage-backed securities	1,872,913
Total	1,974,675	$ 2,238,635
Securities Held to Maturity, Amortized Cost [Abstract]
Due in one year or less	170
Due after one year through five years	40,892
Due after five years through ten years	5,110
Due after ten years	0
Total amortized cost, before mortgage-backed securities	46,172
Mortgage-backed securities	939,673
Securities Held to Maturity, Amortized Cost	985,845	1,056,486
Securities Held to Maturity, Estimated Fair Value [Abstract]
Due in one year or less	170
Due after one year through five years	41,326
Due after five years through ten years	5,241
Due after ten years	0
Total fair value, before mortgage-backed securities	46,737
Mortgage-backed securities	907,901
Total	$ 954,638	$ 1,046,247