Loans - Classification of Loans By Class (Details) - USD ($) $ in Thousands	Sep. 30, 2016	Dec. 31, 2015
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	$ 934,221	$ 875,844
SBA loans held for sale	15,611	13,114
Total loans	949,832	888,958
SBA loans held for investment
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	41,795	39,393
SBA 504 loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	26,067	29,353
Commercial other
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	52,927	49,332
Commercial real estate
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	407,686	391,071
Commercial real estate construction
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	35,395	25,115
Residential mortgage loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	282,317	264,523
Home equity
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	45,495	45,042
Consumer other
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	42,539	32,015
Total consumer loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Total loans held for investment	$ 88,034	$ 77,057