Loans and Asset Quality - Total Loans Held for Investment by Category and Loans Held for Sale (Details) - USD ($) $ in Thousands	Dec. 31, 2021	Dec. 31, 2020
Financing Receivable, Allowance for Credit Loss [Line Items]
Total loans HFI	$ 1,683,832	$ 1,588,446
Total loans HFS	4,290	29,116
Real estate | Commercial real estate
Financing Receivable, Allowance for Credit Loss [Line Items]
Total loans HFI	670,293	556,769
Real estate | One-to-four family residential
Financing Receivable, Allowance for Credit Loss [Line Items]
Total loans HFI	474,420	442,889
Real estate | Construction and development
Financing Receivable, Allowance for Credit Loss [Line Items]
Total loans HFI	106,339	127,321
Commercial and industrial
Financing Receivable, Allowance for Credit Loss [Line Items]
Total loans HFI	311,373	250,428
SBA PPP, net of deferred income
Financing Receivable, Allowance for Credit Loss [Line Items]
Total loans HFI	17,550	118,447
Tax-exempt
Financing Receivable, Allowance for Credit Loss [Line Items]
Total loans HFI	80,726	68,666
Consumer
Financing Receivable, Allowance for Credit Loss [Line Items]
Total loans HFI	$ 23,131	$ 23,926