Securities Available for Sale and Held to Maturity - Contractual Maturities of Available for Sale and Held to Maturity Securities (Details) - USD ($) $ in Thousands	Sep. 30, 2017	Dec. 31, 2016
Securities Available for Sale, Amortized Cost [Abstract]
Due in one year or less	$ 30,117
Due after one year through five years	68,984
Due after five years through ten years	3,947
Due after ten years	34,758
Total amortized cost, before mortgage-backed securities	137,806
Mortgage-backed securities	2,234,449
Total	2,372,255
Securities Available for Sale, Estimated Fair Value [Abstract]
Due in one year or less	30,293
Due after one year through five years	70,611
Due after five years through ten years	3,924
Due after ten years	34,310
Total fair value, before mortgage-backed securities	139,138
Mortgage-backed securities	2,229,951
Total	2,369,089	$ 2,356,682
Securities Held to Maturity, Amortized Cost [Abstract]
Due in one year or less	155
Due after one year through five years	40,620
Due after five years through ten years	8,974
Due after ten years	0
Total amortized cost, before mortgage-backed securities	49,749
Mortgage-backed securities	1,052,534
Securities Held to Maturity, Amortized Cost	1,102,283	1,158,643
Securities Held to Maturity, Estimated Fair Value [Abstract]
Due in one year or less	156
Due after one year through five years	41,883
Due after five years through ten years	9,353
Due after ten years	0
Total fair value, before mortgage-backed securities	51,392
Mortgage-backed securities	1,052,640
Total	$ 1,104,032	$ 1,157,046