Allowance for Loan Losses and Reserve for Unfunded Loan Commitments - Activity in the Allowance for Loan Losses by Portfolio Segment (Details) - USD ($) $ in Thousands	3 Months Ended								12 Months Ended
	Dec. 31, 2016	Sep. 30, 2016	Jun. 30, 2016	Mar. 31, 2016	Dec. 31, 2015	Sep. 30, 2015	Jun. 30, 2015	Mar. 31, 2015	Dec. 31, 2016	Dec. 31, 2015
Allowance for Loan and Lease Losses [Roll Forward]
Beginning Balance				$ 12,759				$ 12,551	$ 12,759	$ 12,551
Charge-offs									(1,463)	(1,448)
Recoveries									63	1,156
Net (charge-offs) recoveries									(1,400)	(292)
Provision for loan losses charged to expense	$ 200	$ 420	$ 400	200	$ 100	$ 200	$ 0	200	1,220	500
Ending Balance	12,579				12,759				12,579	12,759
SBA loans held for investment
Allowance for Loan and Lease Losses [Roll Forward]
Beginning Balance				1,961				1,883	1,961	1,883
Charge-offs									(557)	(370)
Recoveries									33	54
Net (charge-offs) recoveries									(524)	(316)
Provision for loan losses charged to expense									139	394
Ending Balance	1,576				1,961				1,576	1,961
SBA 504 loans
Allowance for Loan and Lease Losses [Roll Forward]
Beginning Balance				741				1,337	741	1,337
Charge-offs									0	(589)
Recoveries									0	0
Net (charge-offs) recoveries									0	(589)
Provision for loan losses charged to expense									(168)	(7)
Ending Balance	573				741				573	741
Commercial loans
Allowance for Loan and Lease Losses [Roll Forward]
Beginning Balance				6,309				6,270	6,309	6,270
Charge-offs									(775)	(309)
Recoveries									29	1,052
Net (charge-offs) recoveries									(746)	743
Provision for loan losses charged to expense									1,166	(704)
Ending Balance	6,729				6,309				6,729	6,309
Residential mortgage loans
Allowance for Loan and Lease Losses [Roll Forward]
Beginning Balance				2,769				2,289	2,769	2,289
Charge-offs									(101)	(50)
Recoveries									0	49
Net (charge-offs) recoveries									(101)	(1)
Provision for loan losses charged to expense									(75)	481
Ending Balance	2,593				2,769				2,593	2,769
Consumer loans
Allowance for Loan and Lease Losses [Roll Forward]
Beginning Balance				817				667	817	667
Charge-offs									(30)	(130)
Recoveries									1	1
Net (charge-offs) recoveries									(29)	(129)
Provision for loan losses charged to expense									137	279
Ending Balance	925				817				925	817
Unallocated
Allowance for Loan and Lease Losses [Roll Forward]
Beginning Balance				$ 162				$ 105	162	105
Charge-offs									0	0
Provision for loan losses charged to expense									21	57
Ending Balance	$ 183				$ 162				$ 183	$ 162