Allowance for Loan Losses and Reserve for Unfunded Loan Commitments - Activity in Allowance for Loan Losses by Portfolio Segment (Details) - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2018	Jun. 30, 2017	Jun. 30, 2018	Jun. 30, 2017
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Beginning Balance	$ 14,196	$ 12,681	$ 13,556	$ 12,579
Charge-offs	(120)	(287)	(206)	(537)
Recoveries	8	6	234	108
Net (charge-offs) recoveries	(112)	(281)	28	(429)
Provision for loan losses charged to expense	550	400	1,050	650
Ending Balance	14,634	12,800	14,634	12,800
SBA loans held for investment
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Beginning Balance	1,504	1,653	1,471	1,576
Charge-offs	(104)	(150)	(184)	(258)
Recoveries	3	3	66	39
Net (charge-offs) recoveries	(101)	(147)	(118)	(219)
Provision for loan losses charged to expense	234	8	284	157
Ending Balance	1,637	1,514	1,637	1,514
Commercial loans
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Beginning Balance	8,139	7,133	7,825	7,302
Charge-offs	0	(120)	0	(196)
Recoveries	4	3	20	56
Net (charge-offs) recoveries	4	(117)	20	(140)
Provision for loan losses charged to expense	212	371	510	225
Ending Balance	8,355	7,387	8,355	7,387
Residential mortgage loans
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Beginning Balance	3,430	2,869	3,130	2,593
Charge-offs	0	0	0	0
Recoveries	0	0	13	12
Net (charge-offs) recoveries	0	0	13	12
Provision for loan losses charged to expense	63	(53)	350	211
Ending Balance	3,493	2,816	3,493	2,816
Consumer loans
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Beginning Balance	1,123	1,026	1,130	925
Charge-offs	(16)	(17)	(22)	(83)
Recoveries	1	0	135	1
Net (charge-offs) recoveries	(15)	(17)	113	(82)
Provision for loan losses charged to expense	41	74	(94)	240
Ending Balance	1,149	1,083	1,149	1,083
Unallocated
Financing Receivable, Allowance for Credit Losses [Roll Forward]
Beginning Balance	0	0	0	183
Charge-offs
Recoveries
Net (charge-offs) recoveries	0	0	0	0
Provision for loan losses charged to expense	0	0	0	(183)
Ending Balance	$ 0	$ 0	$ 0	$ 0