Allowance for Loan Losses and Reserve for Unfunded Loan Commitments - Activity in the Allowance for Loan Losses by Portfolio Segment (Details) - USD ($) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2022	Sep. 30, 2021	Sep. 30, 2022	Sep. 30, 2021
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period	$ 22,858	$ 22,801	$ 22,302	$ 23,105
Charge-offs	(551)	(306)	(1,097)	(1,146)
Recoveries	37	42	130	78
Net recoveries (charge-offs)	(514)	(264)	(967)	(1,068)
Provision for (credit to) loan losses charged to expense	1,517		2,526	500
Balance, end of period	23,861	22,537	23,861	22,537
SBA loans held for investment
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period	758	1,686	1,074	1,301
Charge-offs		(145)		(591)
Recoveries	5		33	34
Net recoveries (charge-offs)	5	(145)	33	(557)
Provision for (credit to) loan losses charged to expense	68	(763)	(276)	34
Balance, end of period	831	778	831	778
Commercial loans
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period	14,908	15,011	15,053	14,992
Charge-offs	(501)	(158)	(1,001)	(551)
Recoveries	23		83	2
Net recoveries (charge-offs)	(478)	(158)	(918)	(549)
Provision for (credit to) loan losses charged to expense	803	983	1,098	1,393
Balance, end of period	15,233	15,836	15,233	15,836
Residential mortgage loans
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period	4,786	4,604	4,114	5,318
Recoveries		42	1	42
Net recoveries (charge-offs)		42	1	42
Provision for (credit to) loan losses charged to expense	449	(540)	1,120	(1,254)
Balance, end of period	5,235	4,106	5,235	4,106
Consumer loans
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period	803	601	671	681
Charge-offs	(50)	(3)	(96)	(4)
Recoveries	9		13
Net recoveries (charge-offs)	(41)	(3)	(83)	(4)
Provision for (credit to) loan losses charged to expense	349	151	523	72
Balance, end of period	1,111	749	1,111	749
Residential construction loans
Allowance for Loan and Lease Losses [Roll Forward]
Balance, beginning of period	1,603	899	1,390	813
Provision for (credit to) loan losses charged to expense	(152)	169	61	255
Balance, end of period	$ 1,451	$ 1,068	$ 1,451	$ 1,068