Loans - Classification of Loans By Class (Details) - USD ($) $ in Thousands	Mar. 31, 2017	Dec. 31, 2016
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	$ 988,514	$ 958,641
SBA loans held for sale	12,163	14,773
Total loans	1,000,677	973,414
Residential mortgage loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	305,578	289,093
Consumer loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	96,084	91,541
SBA loans held for investment
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	42,403	42,492
SBA 504 loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	25,111	26,344
Commercial loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	519,338	509,171
Commercial loans | Other
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	60,381	58,447
Commercial loans | Commercial real estate
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	425,581	422,418
Commercial loans | Commercial real estate construction
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	33,376	28,306
Residential mortgage loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	305,578	289,093
Consumer loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	96,084	91,541
Consumer loans | Other
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	46,789	44,130
Consumer loans | Home equity
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	$ 49,295	$ 47,411