Securities Available for Sale and Held to Maturity - Contractual Maturities of Available for Sale and Held to Maturity Securities (Details) - USD ($) $ in Thousands	Jun. 30, 2017	Dec. 31, 2016
Securities Available for Sale, Amortized Cost [Abstract]
Due in one year or less	$ 28,442
Due after one year through five years	75,894
Due after five years through ten years	6,278
Due after ten years	35,941
Total amortized cost, before mortgage-backed securities	146,555
Mortgage-backed securities	2,303,750
Total	2,450,305
Securities Available for Sale, Estimated Fair Value [Abstract]
Due in one year or less	28,649
Due after one year through five years	77,675
Due after five years through ten years	6,280
Due after ten years	35,459
Total fair value, before mortgage-backed securities	148,063
Mortgage-backed securities	2,299,625
Total	2,447,688	$ 2,356,682
Securities Held to Maturity, Amortized Cost [Abstract]
Due in one year or less	155
Due after one year through five years	39,820
Due after five years through ten years	9,792
Due after ten years	0
Total amortized cost, before mortgage-backed securities	49,767
Mortgage-backed securities	1,089,987
Securities Held to Maturity, Amortized Cost	1,139,754	1,158,643
Securities Held to Maturity, Estimated Fair Value [Abstract]
Due in one year or less	155
Due after one year through five years	41,165
Due after five years through ten years	10,148
Due after ten years	0
Total fair value, before mortgage-backed securities	51,468
Mortgage-backed securities	1,090,817
Total	$ 1,142,285	$ 1,157,046