Loans and Asset Quality - Total Loans Held for Investment by Category and Loans Held for Sale (Details) - USD ($) $ in Thousands	Sep. 30, 2020	Dec. 31, 2019
Financing Receivable, Allowance for Credit Loss [Line Items]
Loans held for investment	$ 1,649,272	$ 1,438,924
Loans held for sale	23,358	5,089
Real estate | Commercial real estate
Financing Receivable, Allowance for Credit Loss [Line Items]
Loans held for investment	567,037	531,990
Real estate | One-to-four family residential
Financing Receivable, Allowance for Credit Loss [Line Items]
Loans held for investment	426,758	420,020
Real estate | Construction and development
Financing Receivable, Allowance for Credit Loss [Line Items]
Loans held for investment	129,879	132,461
Commercial and industrial
Financing Receivable, Allowance for Credit Loss [Line Items]
Loans held for investment	249,313	267,940
SBA PPP, net of deferred income
Financing Receivable, Allowance for Credit Loss [Line Items]
Loans held for investment	193,532	0
Tax-exempt
Financing Receivable, Allowance for Credit Loss [Line Items]
Loans held for investment	59,418	56,494
Consumer
Financing Receivable, Allowance for Credit Loss [Line Items]
Loans held for investment	$ 23,335	$ 30,019