Allowance for Loan Losses and Reserve for Unfunded Loan Commitments - Activity in Allowance for Loan Losses by Portfolio Segment (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2016	Jun. 30, 2015	Jun. 30, 2016	Jun. 30, 2015
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Beginning Balance	$ 12,634	$ 12,181	$ 12,759	$ 12,551
Charge-offs	(294)	(160)	(636)	(1,008)
Recoveries	18	383	35	661
Net (charge-offs) recoveries	(276)	223	(601)	(347)
Provision for loan losses charged to expense	400	0	600	200
Ending Balance	12,758	12,404	12,758	12,404
SBA loans held for investment
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Beginning Balance	1,713	1,873	1,961	1,883
Charge-offs	(142)	(6)	(228)	(135)
Recoveries	4	2	15	40
Net (charge-offs) recoveries	(138)	(4)	(213)	(95)
Provision for loan losses charged to expense	55	129	(118)	210
Ending Balance	1,630	1,998	1,630	1,998
SBA 504 loans
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Beginning Balance	598	1,162	741	1,337
Charge-offs	0	0	0	(589)
Recoveries	0	0	0	0
Net (charge-offs) recoveries	0	0	0	(589)
Provision for loan losses charged to expense	(10)	(283)	(153)	131
Ending Balance	588	879	588	879
Total commercial loans
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Beginning Balance	6,312	5,624	6,309	6,270
Charge-offs	(152)	(147)	(380)	(247)
Recoveries	13	370	19	571
Net (charge-offs) recoveries	(139)	223	(361)	324
Provision for loan losses charged to expense	393	212	618	(535)
Ending Balance	6,566	6,059	6,566	6,059
Residential mortgage loans
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Beginning Balance	2,892	2,341	2,769	2,289
Charge-offs	0	0	0	0
Recoveries	0	10	0	49
Net (charge-offs) recoveries	0	10	0	49
Provision for loan losses charged to expense	72	194	195	207
Ending Balance	2,964	2,545	2,964	2,545
Total consumer loans
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Beginning Balance	836	710	817	667
Charge-offs	0	(7)	(28)	(37)
Recoveries	1	1	1	1
Net (charge-offs) recoveries	1	(6)	(27)	(36)
Provision for loan losses charged to expense	24	52	71	125
Ending Balance	861	756	861	756
Unallocated
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Beginning Balance	283	471	162	105
Charge-offs
Recoveries
Net (charge-offs) recoveries	0	0	0	0
Provision for loan losses charged to expense	(134)	(304)	(13)	62
Ending Balance	$ 149	$ 167	$ 149	$ 167