Loans and Allowance for Credit Losses on Loans - Loans Balance (Details) - USD ($)	Dec. 31, 2022	Dec. 31, 2021	Dec. 31, 2020	Jan. 01, 2020	Dec. 31, 2019
Loans held-for-investment:
Total loan balance	$ 3,299,067,000	$ 3,090,788,000
Deferred loan fees, net	(517,000)	(3,462,000)
Loans, net of deferred fees	3,298,550,000	3,087,326,000	$ 2,619,261,000
Allowance for credit losses on loans	(47,512,000)	(43,290,000)	(44,400,000)	$ (31,855,000)	$ (23,285,000)
Loans, net	3,251,038,000	3,044,036,000
Commercial. | SBA Paycheck Protection Program ("PPP")
Loans held-for-investment:
Allowance for credit losses on loans		0
Commercial
Loans held-for-investment:
Total loan balance	533,915,000	682,834,000
Allowance for credit losses on loans	(6,617,000)	(8,414,000)	(11,587,000)	(6,790,000)	(10,453,000)
Commercial | Commercial.
Loans held-for-investment:
Total loan balance	533,915,000	682,834,000
SBA Paycheck Protection Program ("PPP") loans
Loans held-for-investment:
Total loan balance	1,166,000
Loans, net		88,726,000
CRE - Owner Occupied
Loans held-for-investment:
Total loan balance	614,663,000	595,934,000
Allowance for credit losses on loans	(5,751,000)	(7,954,000)	(8,560,000)	(6,994,000)	(3,825,000)
CRE - Owner Occupied | Real estate
Loans held-for-investment:
Total loan balance	614,663,000	595,934,000
CRE - Non-Owner Occupied
Loans held-for-investment:
Total loan balance	1,066,368,000	902,326,000
Allowance for credit losses on loans	(22,135,000)	(17,125,000)	(16,416,000)	(11,672,000)	(3,760,000)
CRE - Non-Owner Occupied | Real estate
Loans held-for-investment:
Total loan balance	1,066,368,000	902,326,000
Land & Construction
Loans held-for-investment:
Total loan balance	163,577,000	147,855,000
Allowance for credit losses on loans	(2,941,000)	(1,831,000)	(2,509,000)	(1,458,000)	(2,621,000)
Land & Construction | Real estate
Loans held-for-investment:
Total loan balance	163,577,000	147,855,000
Home Equity
Loans held-for-investment:
Total loan balance	120,724,000	109,579,000
Allowance for credit losses on loans	(666,000)	(864,000)	(1,297,000)	(1,321,000)	(2,244,000)
Home Equity | Real estate
Loans held-for-investment:
Total loan balance	120,724,000	109,579,000
Multi-Family
Loans held-for-investment:
Total loan balance	244,882,000	218,856,000
Allowance for credit losses on loans	(3,366,000)	(2,796,000)	(2,804,000)	(1,253,000)	(57,000)
Multi-Family | Real estate
Loans held-for-investment:
Total loan balance	244,882,000	218,856,000
Residential Mortgage
Loans held-for-investment:
Total loan balance	537,905,000	416,660,000
Allowance for credit losses on loans	(5,907,000)	(4,132,000)	(943,000)	(678,000)	(243,000)
Residential Mortgage | Real estate
Loans held-for-investment:
Total loan balance	537,905,000	416,660,000
Consumer and Other
Loans held-for-investment:
Total loan balance	17,033,000	16,744,000
Allowance for credit losses on loans	(129,000)	(174,000)	$ (284,000)	$ (1,689,000)	(82,000)
Consumer and Other | Consumer
Loans held-for-investment:
Total loan balance	$ 17,033,000	$ 16,744,000
ASU 2016-03 - Topic 326
Loans held-for-investment:
Allowance for credit losses on loans					(8,570,000)
ASU 2016-03 - Topic 326 | Commercial
Loans held-for-investment:
Allowance for credit losses on loans					3,663,000
ASU 2016-03 - Topic 326 | CRE - Owner Occupied
Loans held-for-investment:
Allowance for credit losses on loans					(3,169,000)
ASU 2016-03 - Topic 326 | CRE - Non-Owner Occupied
Loans held-for-investment:
Allowance for credit losses on loans					(7,912,000)
ASU 2016-03 - Topic 326 | Land & Construction
Loans held-for-investment:
Allowance for credit losses on loans					1,163,000
ASU 2016-03 - Topic 326 | Home Equity
Loans held-for-investment:
Allowance for credit losses on loans					923,000
ASU 2016-03 - Topic 326 | Multi-Family
Loans held-for-investment:
Allowance for credit losses on loans					(1,196,000)
ASU 2016-03 - Topic 326 | Residential Mortgage
Loans held-for-investment:
Allowance for credit losses on loans					(435,000)
ASU 2016-03 - Topic 326 | Consumer and Other
Loans held-for-investment:
Allowance for credit losses on loans					$ (1,607,000)