Loans - Classification of Loans By Class (Details) - USD ($) $ in Thousands	Sep. 30, 2020	Dec. 31, 2019
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	$ 1,607,099	$ 1,412,029
SBA loans held for sale	6,192	13,529
Total loans	1,613,291	1,425,558
SBA 504 loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	21,811	26,726
Residential mortgage loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	473,420	467,706
Consumer loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	148,086	143,524
SBA loans held for investment
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	47,125	35,767
SBA PPP loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	138,895	0
Commercial loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	799,573	765,032
Commercial loans | Other
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	115,715	112,014
Commercial loans | Commercial real estate
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	596,243	578,643
Commercial loans | Commercial real estate construction
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	65,804	47,649
Residential mortgage loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	473,420	467,706
Consumer loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	148,086	143,524
Consumer loans | Other
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	80,829	73,935
Consumer loans | Home equity
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	$ 67,257	$ 69,589