Loans - Aging Analysis of Past Due and Nonaccrual Loans by Class (Details) - USD ($) $ in Thousands	Sep. 30, 2016	Dec. 31, 2015
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, nonaccrual status	$ 6,527	$ 7,260
Loans receivable, past due	12,451	13,628
Loan receivable, current	937,381	875,330
Total loans	934,221	875,844
SBA loans held for sale	15,611	13,114
Total loans	949,832	888,958
Performing troubled debt restructurings	666	3,000
Troubled Debt Restructuring (TDR) | Non Accrual
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, nonaccrual status	154	293
SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, nonaccrual status	1,967	1,764
Loans receivable, past due	2,420	3,373
Loan receivable, current	39,375	36,020
Total loans	41,795	39,393
SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, nonaccrual status	513	518
Loans receivable, past due	513	518
Loan receivable, current	25,554	28,835
Total loans	26,067	29,353
Commercial other
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, nonaccrual status	611	10
Loans receivable, past due	611	167
Loan receivable, current	52,316	49,165
Total loans	52,927	49,332
Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, nonaccrual status	43	2,154
Loans receivable, past due	1,751	2,881
Loan receivable, current	405,935	388,190
Total loans	407,686	391,071
Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, nonaccrual status	306	0
Loans receivable, past due	306	356
Loan receivable, current	35,089	24,759
Total loans	35,395	25,115
Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, nonaccrual status	2,598	2,224
Loans receivable, past due	6,037	5,609
Loan receivable, current	276,280	258,914
Total loans	282,317	264,523
Home equity
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, nonaccrual status	489	590
Loans receivable, past due	813	723
Loan receivable, current	44,682	44,319
Total loans	45,495	45,042
Consumer other
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, nonaccrual status	0	0
Loans receivable, past due	0	1
Loan receivable, current	42,539	32,014
Total loans	42,539	32,015
Loans Held for Investment
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, nonaccrual status	6,527	7,260
Loans receivable, past due	12,451	13,628
Loan receivable, current	921,770	862,216
Total loans	934,221	875,844
SBA loans held for sale
Financing Receivable, Recorded Investment [Line Items]
Loan receivable, current	15,611	13,114
SBA loans held for sale	15,611	13,114
Loans Guaranteed By SBA | Non Accrual
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, nonaccrual status	624	288
30-59 days past due
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	4,644	4,548
30-59 days past due | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	445	1,153
30-59 days past due | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	0	0
30-59 days past due | Commercial other
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	0	157
30-59 days past due | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	1,708	444
30-59 days past due | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	0	356
30-59 days past due | Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	2,491	2,307
30-59 days past due | Home equity
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	0	130
30-59 days past due | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	0	1
30-59 days past due | Loans Held for Investment
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	4,644	4,548
60-89 days past due
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	1,280	1,820
60-89 days past due | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	8	456
60-89 days past due | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	0	0
60-89 days past due | Commercial other
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	0	0
60-89 days past due | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	0	283
60-89 days past due | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	0	0
60-89 days past due | Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	948	1,078
60-89 days past due | Home equity
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	324	3
60-89 days past due | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	0	0
60-89 days past due | Loans Held for Investment
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	1,280	1,820
90 days and still accruing
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	0	0
90 days and still accruing | SBA loans held for investment
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	0	0
90 days and still accruing | SBA 504 loans
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	0	0
90 days and still accruing | Commercial other
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	0	0
90 days and still accruing | Commercial real estate
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	0	0
90 days and still accruing | Commercial real estate construction
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	0	0
90 days and still accruing | Residential mortgage loans
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	0	0
90 days and still accruing | Home equity
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	0	0
90 days and still accruing | Consumer other
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	0	0
90 days and still accruing | Loans Held for Investment
Financing Receivable, Recorded Investment [Line Items]
Loans receivable, past due	$ 0	$ 0