Loans and Asset Quality - Summary of Loans by Risk Rating (Details) - USD ($) $ in Thousands	Sep. 30, 2020	Dec. 31, 2019
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	$ 1,649,272	$ 1,438,924
Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	1,630,977	1,397,747
Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	1,286	28,182
Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	17,009	12,995
Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	0	0
Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	0	0
Real estate | Commercial real estate
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	567,037	531,990
Real estate | Commercial real estate | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	562,169	515,926
Real estate | Commercial real estate | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	582	14,118
Real estate | Commercial real estate | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	4,286	1,946
Real estate | Commercial real estate | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	0	0
Real estate | Commercial real estate | Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	0	0
Real estate | One-to-four family residential
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	426,758	420,020
Real estate | One-to-four family residential | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	424,921	416,884
Real estate | One-to-four family residential | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	492	2,021
Real estate | One-to-four family residential | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	1,345	1,115
Real estate | One-to-four family residential | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	0	0
Real estate | One-to-four family residential | Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	0	0
Real estate | Construction and development
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	129,879	132,461
Real estate | Construction and development | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	129,100	131,185
Real estate | Construction and development | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	0	565
Real estate | Construction and development | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	779	711
Real estate | Construction and development | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	0	0
Real estate | Construction and development | Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	0	0
Commercial and industrial
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	249,313	267,940
Commercial and industrial | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	238,616	247,382
Commercial and industrial | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	212	11,473
Commercial and industrial | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	10,485	9,085
Commercial and industrial | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	0	0
Commercial and industrial | Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	0	0
SBA PPP, net of deferred income
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	193,532	0
SBA PPP, net of deferred income | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	193,532	0
SBA PPP, net of deferred income | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	0	0
SBA PPP, net of deferred income | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	0	0
SBA PPP, net of deferred income | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	0	0
SBA PPP, net of deferred income | Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	0	0
Tax-exempt
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	59,418	56,494
Tax-exempt | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	59,418	56,494
Tax-exempt | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	0	0
Tax-exempt | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	0	0
Tax-exempt | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	0	0
Tax-exempt | Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	0	0
Consumer
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	23,335	30,019
Consumer | Pass
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	23,221	29,876
Consumer | Special Mention
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	0	5
Consumer | Substandard
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	114	138
Consumer | Doubtful
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	0	0
Consumer | Loss
Financing Receivable, Credit Quality Indicator [Line Items]
Loans held for investment	$ 0	$ 0