Commitments, Contingencies, and Concentrations of Credit Risk - Schedule of Maximum Credit Risk for Securities (Details) $ in Thousands	Dec. 31, 2020 USD ($)
Fannie Mae – mortgage-backed securities
Loss Contingencies [Line Items]
Amortized cost, Available for sale and held to maturity	$ 571,245
Fair value, Available for sale and held to maturity	585,035
Freddie Mac – mortgage-backed securities
Loss Contingencies [Line Items]
Amortized cost, Available for sale and held to maturity	234,780
Fair value, Available for sale and held to maturity	237,159
Ginnie Mae – mortgage-backed securities
Loss Contingencies [Line Items]
Amortized cost, Available for sale and held to maturity	549,811
Fair value, Available for sale and held to maturity	552,830
Federal Farm Credit Bank – bonds
Loss Contingencies [Line Items]
Amortized cost, Available for sale and held to maturity	22,150
Fair value, Available for sale and held to maturity	22,436
Small Business Administration securities
Loss Contingencies [Line Items]
Amortized cost, Available for sale and held to maturity	17,671
Fair value, Available for sale and held to maturity	17,671
Federal Reserve Bank  - common stock
Loss Contingencies [Line Items]
Amortized cost, Available for sale and held to maturity	5,855
Fair value, Available for sale and held to maturity	5,855
First Citizens Bank – corporate bonds
Loss Contingencies [Line Items]
Amortized cost, Available for sale and held to maturity	11,000
Fair value, Available for sale and held to maturity	10,999
Bank of America corporate bonds
Loss Contingencies [Line Items]
Amortized cost, Available for sale and held to maturity	7,000
Fair value, Available for sale and held to maturity	7,409
Citigroup, Inc. corporate bonds
Loss Contingencies [Line Items]
Amortized cost, Available for sale and held to maturity	6,014
Fair value, Available for sale and held to maturity	6,346
Federal Home Loan Bank of Atlanta -  common stock
Loss Contingencies [Line Items]
Amortized cost, Available for sale and held to maturity	5,037
Fair value, Available for sale and held to maturity	5,319
Loudoun County, Virginia - municipal bond
Loss Contingencies [Line Items]
Amortized cost, Available for sale and held to maturity	5,009
Fair value, Available for sale and held to maturity	5,294
Goldman Sachs Group Inc. corporate bond
Loss Contingencies [Line Items]
Amortized cost, Available for sale and held to maturity	40,015
Fair value, Available for sale and held to maturity	$ 40,356