Securities Available for Sale and Held to Maturity - Contractual Maturities of Available for Sale and Held to Maturity Securities (Details) - USD ($) $ in Thousands	Dec. 31, 2019	Dec. 31, 2018
Securities Available for Sale, Amortized Cost [Abstract]
Due in one year or less	$ 22,480
Due after one year through five years	1,598
Due after five years through ten years	2,561
Due after ten years	21,278
Total amortized cost, before mortgage-backed securities	47,917
Mortgage-backed securities	1,553,067
Securities Available for Sale, Amortized Cost	1,600,984	$ 1,854,529
Securities Available for Sale, Estimated Fair Value [Abstract]
Due in one year or less	22,623
Due after one year through five years	1,614
Due after five years through ten years	2,513
Due after ten years	21,042
Total fair value, before mortgage-backed securities	47,792
Mortgage-backed securities	1,554,612
Total	1,602,404	1,811,813
Securities Held to Maturity, Amortized Cost [Abstract]
Due in one year or less	5,071
Due after one year through five years	30,491
Due after five years through ten years	0
Due after ten years	0
Total amortized cost, before mortgage-backed securities	35,562
Mortgage-backed securities	702,537
Securities Held to Maturity, Amortized Cost	738,099	909,643
Securities Held to Maturity, Estimated Fair Value [Abstract]
Due in one year or less	5,115
Due after one year through five years	31,048
Due after five years through ten years	0
Due after ten years	0
Total fair value, before mortgage-backed securities	36,163
Mortgage-backed securities	710,039
Total	$ 746,202	$ 889,733