Allowance for Loan Losses and Reserve for Unfunded Loan Commitments - Activity in Allowance for Loan Losses by Portfolio Segment (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2016	Sep. 30, 2015	Sep. 30, 2016	Sep. 30, 2015
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Beginning Balance	$ 12,758	$ 12,404	$ 12,759	$ 12,551
Charge-offs	(516)	(198)	(1,153)	(1,205)
Recoveries	23	15	59	675
Net (charge-offs) recoveries	(493)	(183)	(1,094)	(530)
Provision for loan losses charged to expense	420	200	1,020	400
Ending Balance	12,685	12,421	12,685	12,421
SBA loans held for investment
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Beginning Balance	1,630	1,998	1,961	1,883
Charge-offs	(140)	(86)	(369)	(220)
Recoveries	17	10	33	49
Net (charge-offs) recoveries	(123)	(76)	(336)	(171)
Provision for loan losses charged to expense	128	(11)	10	199
Ending Balance	1,635	1,911	1,635	1,911
SBA 504 loans
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Beginning Balance	588	879	741	1,337
Charge-offs	0	0	0	(589)
Recoveries	0	0	0	0
Net (charge-offs) recoveries	0	0	0	(589)
Provision for loan losses charged to expense	(21)	(89)	(174)	42
Ending Balance	567	790	567	790
Total commercial loans
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Beginning Balance	6,566	6,059	6,309	6,270
Charge-offs	(376)	(10)	(756)	(257)
Recoveries	6	5	25	576
Net (charge-offs) recoveries	(370)	(5)	(731)	319
Provision for loan losses charged to expense	616	146	1,234	(389)
Ending Balance	6,812	6,200	6,812	6,200
Residential mortgage loans
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Beginning Balance	2,964	2,545	2,769	2,289
Charge-offs	0	(50)	0	(50)
Recoveries	0	0	0	49
Net (charge-offs) recoveries	0	(50)	0	(1)
Provision for loan losses charged to expense	(448)	186	(253)	393
Ending Balance	2,516	2,681	2,516	2,681
Total consumer loans
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Beginning Balance	861	756	817	667
Charge-offs	0	(52)	(28)	(89)
Recoveries	0	0	1	1
Net (charge-offs) recoveries	0	(52)	(27)	(88)
Provision for loan losses charged to expense	31	84	102	209
Ending Balance	892	788	892	788
Unallocated
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Beginning Balance	149	167	162	105
Charge-offs	0	0	0	0
Recoveries	0	0	0	0
Net (charge-offs) recoveries	0	0	0	0
Provision for loan losses charged to expense	114	(116)	101	(54)
Ending Balance	$ 263	$ 51	$ 263	$ 51