Loans - Classification of Loans By Class (Details) - USD ($) $ in Thousands	Jun. 30, 2018	Dec. 31, 2017
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	$ 1,231,622	$ 1,147,864
SBA loans held for sale	14,889	22,810
Total loans	1,246,511	1,170,674
SBA loans held for investment
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	41,351	43,999
Residential mortgage loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	398,383	365,145
SBA loans held for investment
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	41,351	43,999
SBA 504 loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	23,927	21,871
Commercial loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	673,480	628,865
Commercial loans | Other
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	96,759	82,825
Commercial loans | Commercial real estate
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	479,530	469,696
Commercial loans | Commercial real estate construction
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	73,264	54,473
Residential mortgage loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	398,383	365,145
Consumer loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	118,408	109,855
Consumer loans | Other
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	57,434	54,038
Consumer loans | Home equity
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	$ 60,974	$ 55,817