Securities Available for Sale and Held to Maturity - Contractual Maturities of Available for Sale and Held to Maturity Securities (Details) - USD ($) $ in Thousands	Mar. 31, 2020	Dec. 31, 2019
Securities Available for Sale, Amortized Cost [Abstract]
Due in one year or less	$ 20,505
Due after one year through five years	1,304
Due after five years through ten years	2,499
Due after ten years	20,480
Total amortized cost, before mortgage-backed securities	44,788
Mortgage-backed securities	1,747,064
Securities Available for Sale, Amortized Cost	1,791,852	$ 1,600,984
Securities Available for Sale, Estimated Fair Value [Abstract]
Due in one year or less	20,617
Due after one year through five years	1,317
Due after five years through ten years	2,456
Due after ten years	20,372
Total fair value, before mortgage-backed securities	44,762
Mortgage-backed securities	1,789,017
Total	1,833,779	1,602,404
Securities Held to Maturity, Amortized Cost [Abstract]
Due in one year or less	5,079
Due after one year through five years	26,679
Due after five years through ten years	0
Due after ten years	0
Total amortized cost, before mortgage-backed securities	31,758
Mortgage-backed securities	672,518
Securities Held to Maturity, Amortized Cost	704,276	738,099
Securities Held to Maturity, Estimated Fair Value [Abstract]
Due in one year or less	5,115
Due after one year through five years	26,985
Due after five years through ten years	0
Due after ten years	0
Total fair value, before mortgage-backed securities	32,100
Mortgage-backed securities	699,341
Total	$ 731,441	$ 746,202