Securities Available for Sale and Held to Maturity - Contractual Maturities of Available for Sale and Held to Maturity Securities (Details) - USD ($) $ in Thousands	Mar. 31, 2017	Dec. 31, 2016
Securities Available for Sale, Amortized Cost [Abstract]
Due in one year or less	$ 31,880
Due after one year through five years	85,618
Due after five years through ten years	6,881
Due after ten years	37,507
Total amortized cost, before mortgage-backed securities	161,886
Mortgage-backed securities	2,210,869
Total	2,372,755
Securities Available for Sale, Estimated Fair Value [Abstract]
Due in one year or less	32,032
Due after one year through five years	87,786
Due after five years through ten years	6,880
Due after ten years	36,718
Total fair value, before mortgage-backed securities	163,416
Mortgage-backed securities	2,202,138
Total	2,365,554	$ 2,356,682
Securities Held to Maturity, Amortized Cost [Abstract]
Due in one year or less	145
Due after one year through five years	36,380
Due after five years through ten years	13,406
Due after ten years	0
Total amortized cost, before mortgage-backed securities	49,931
Mortgage-backed securities	1,106,136
Securities Held to Maturity, Amortized Cost	1,156,067	1,158,643
Securities Held to Maturity, Estimated Fair Value [Abstract]
Due in one year or less	145
Due after one year through five years	37,765
Due after five years through ten years	13,851
Due after ten years	0
Total fair value, before mortgage-backed securities	51,761
Mortgage-backed securities	1,102,654
Total	$ 1,154,415	$ 1,157,046