Allowance for Loan Losses and Reserve for Unfunded Loan Commitments - Activity in Allowance for Loan Losses (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2020	Sep. 30, 2019	Sep. 30, 2020	Sep. 30, 2019
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning Balance	$ 20,234	$ 15,965	$ 16,395	$ 15,488
Charge-offs	(1)	(729)	(745)	(1,124)
Recoveries	4	16	587	38
Net (charge-offs) recoveries	3	(713)	(158)	(1,086)
Provision for (credit to) loan losses charged to expense	2,000	750	6,000	1,600
Ending Balance	22,237	16,002	22,237	16,002
SBA loans held for investment
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning Balance	1,004	1,321	1,079	1,655
Charge-offs	(1)	(99)	(27)	(492)
Recoveries	3	13	83	16
Net (charge-offs) recoveries	2	(86)	56	(476)
Provision for (credit to) loan losses charged to expense	388	(68)	259	(12)
Ending Balance	1,394	1,167	1,394	1,167
Commercial loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning Balance	12,336	9,144	9,722	8,705
Charge-offs		(500)	(518)	(501)
Recoveries	1	3	504	12
Net (charge-offs) recoveries	1	(497)	(14)	(489)
Provision for (credit to) loan losses charged to expense	963	778	3,592	1,209
Ending Balance	13,300	9,425	13,300	9,425
Residential mortgage loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning Balance	5,439	4,198	4,254	3,900
Charge-offs		(130)	(200)	(130)
Net (charge-offs) recoveries		(130)	(200)	(130)
Provision for (credit to) loan losses charged to expense	485	52	1,870	350
Ending Balance	5,924	4,120	5,924	4,120
Consumer loans
Financing Receivable, Allowance for Credit Loss [Roll Forward]
Beginning Balance	1,455	1,302	1,340	1,228
Charge-offs				(1)
Recoveries				10
Net (charge-offs) recoveries				9
Provision for (credit to) loan losses charged to expense	164	(12)	279	53
Ending Balance	$ 1,619	$ 1,290	$ 1,619	$ 1,290