Allowance for Loan Losses and Reserve for Unfunded Loan Commitments - Activity in the Allowance for Loan Losses by Portfolio Segment (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2021	Sep. 30, 2020	Sep. 30, 2021	Sep. 30, 2020
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period	$ 22,801	$ 20,234	$ 23,105	$ 16,395
Charge-offs	(306)	(1)	(1,146)	(745)
Recoveries	42	4	78	587
Net (charge-offs) recoveries	(264)	3	(1,068)	(158)
Provision for (credit to) loan losses charged to expense		2,000	500	6,000
Balance, end of period	22,537	22,237	22,537	22,237
SBA loans held for investment
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period	1,686	1,004	1,301	1,079
Charge-offs	(145)	(1)	(591)	(27)
Recoveries		3	34	83
Net (charge-offs) recoveries	(145)	2	(557)	56
Provision for (credit to) loan losses charged to expense	(763)	388	34	259
Balance, end of period	778	1,394	778	1,394
Commercial loans
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period	15,011	12,336	14,992	9,722
Charge-offs	(158)		(551)	(518)
Recoveries		1	2	504
Net (charge-offs) recoveries	(158)	1	(549)	(14)
Provision for (credit to) loan losses charged to expense	983	963	1,393	3,592
Balance, end of period	15,836	13,300	15,836	13,300
Residential mortgage loans
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period	4,604	5,439	5,318	4,254
Charge-offs				(200)
Recoveries	42		42
Net (charge-offs) recoveries	42		42	(200)
Provision for (credit to) loan losses charged to expense	(540)	485	(1,254)	1,870
Balance, end of period	4,106	5,924	4,106	5,924
Consumer loans
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period	601	611	681	681
Charge-offs	(3)		(4)
Net (charge-offs) recoveries	(3)		(4)
Provision for (credit to) loan losses charged to expense	151	66	72	(4)
Balance, end of period	749	677	749	677
Residential construction loans
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period	899	844	813	659
Provision for (credit to) loan losses charged to expense	169	98	255	283
Balance, end of period	$ 1,068	$ 942	$ 1,068	$ 942