Consolidated Statements of Cash Flows - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2023	Dec. 31, 2022	Dec. 31, 2021
Cash flows from operating activities:
Net (loss) income from continuing operations	$ (51,773)	$ 16,997	$ 52,624
Net income (loss) from discontinued operations	0	337	(144)
Adjustments to reconcile net (loss) income to net cash provided by (used in) operating activities:
Goodwill impairment	26,826	0	0
Depreciation and amortization	1,696	1,977	2,144
Deferred income tax (benefit) expense	(7,055)	(20)	1,923
Provision for credit losses - loans	24,703	25,687	117
Benefit for credit losses - unfunded commitments	(2,380)	0	0
Accretion of fair value adjustments (discounts) on acquired loans	(2,611)	(7,392)	(2,006)
Accretion of fair value adjustments (premiums) on acquired time deposits and acquired subordinated notes	(898)	(1,581)	(3,401)
Proceeds from sale of mortgage loans held for sale	251,797	522,252	1,228,021
Mortgage loans held for sale, originated	(248,721)	(416,772)	(1,171,787)
Gain on sale of mortgage loans	(2,098)	(4,451)	(21,432)
Proceeds from sale of guaranteed government loans held for sale	82,524	54,271	0
Guaranteed government loans held for sale, originated	(36,661)	(88,077)	0
Gain on sale of guaranteed government loans	(5,704)	(4,734)	0
Gain on sale of Paycheck Protection Program loans	0	0	(24,315)
Loss (gain) on disposal of premises and equipment and other assets	530	(263)	110
Realized loss on sale of available for sale securities	649	0	144
Realized loss on sale of other equity securities and other investments	1,523	0	0
Investment amortization expense, net	653	1,332	1,865
Amortization of subordinated debt issuance costs	35	35	206
Intangible amortization	1,295	1,525	1,671
Fair value adjustments of other equity investments	110	(9,306)	(7,316)
Fair value adjustments attributable to mortgage servicing rights	2,887	(2,248)	0
Fair value adjustments on other real estate owned	0	0	75
Increase (decrease) in cash surrender value of bank owned life insurance	1,195	(1,348)	(932)
Increase in accrued interest receivable	(3,398)	(1,996)	(4,145)
(Increase) decrease in other assets	(16,632)	2,389	16,492
Increase (decrease) in other liabilities	25,564	5,262	(10,921)
Net cash provided by operating activities - continuing operations	44,056	93,876	58,993
Net cash provided by operating activities - discontinued operations	0	55	220
Cash provided by (used in) operating activities	44,056	93,931	59,213
Cash Flows From Investing Activities
Net increase in loans held for investment	(70,125)	(631,797)	(59,053)
Net (increase) decrease in federal funds sold	(3,025)	42,477	(41,396)
Purchases of securities available for sale	0	(68,261)	(264,929)
Proceeds from calls, sales, paydowns, and maturities of securities available for sale	32,708	32,655	71,804
Proceeds from sale of other equity investments and other investments	7,854	0	0
Proceeds from sale of other real estate owned	264	70	341
Proceeds from sale of Paycheck Protection Program loans	0	0	705,930
Net decrease (increase) in Paycheck Protection Program loans	9,581	18,439	(382,830)
Net change in restricted equity and other investments	2,682	(13,899)	(78)
Purchase of premises and equipment	(961)	(455)	(1,217)
Proceeds from sale of premises and equipment and other assets	1,016	2,211	547
Purchase of bank owned life insurance	0	0	(9,600)
Redemption of bank owned life insurance	0	414	0
Proceeds from sale of LSMG	250	0	0
Capital calls of small business investment company funds and other investments	(5,128)	(11,310)	(11,582)
Net cash acquired in acquisition of Bay Banks of Virginia, Inc.	0	0	44,066
Nonincome distributions from limited liability companies	1,671	1,028	647
Net cash (used in) provided by investing activities - continuing operations	(23,213)	(628,428)	52,650
Net cash provided by (used in) investing activities - discontinued operations	0	245	(166)
Cash (used in) provided by investing activities	(23,213)	(628,183)	52,484
Cash flows from financing activities:
Net (decrease) increase in demand, savings, and other interest-bearing deposits	(436,839)	312,277	452,173
Net increase (decrease) in time deposits	501,162	(106,061)	(127,174)
Common stock dividends paid	(4,641)	(9,175)	(7,183)
FHLB advances	1,940,000	822,900	721,000
FHLB repayments	(2,041,700)	(521,200)	(836,000)
FRB advances	65,000	0	434,336
FRB repayments	(51)	(17,850)	(722,900)
Stock option exercises	26	14	804
Dividend reinvestment plan issuances	77	73	0
Redemptions of subordinated notes	0	0	(14,150)
Net cash provided by (used in) financing activities - continuing operations	23,034	480,978	(99,094)
Net cash provided by financing activities - discontinued operations	0	0	0
Cash provided by (used in) financing activities	23,034	480,978	(99,094)
Net increase (decrease) in cash and due from banks	43,877	(53,274)	12,603
Cash and due from banks at beginning of period	77,274	130,548	117,945
Cash and due from banks and restricted cash at end of period	121,151	77,274	130,548
Supplemental Schedule of Cash Flow Information
Interest	67,868	16,011	11,583
Income taxes	6,656	2,077	10,131
Non-cash investing and financing activities:
Unrealized losses on securities available for sale	(592)	(53,405)	(6,024)
Restricted stock awards, net of forfeitures	1,573	1,564	1,331
Assets acquired in business combination	0	0	1,224,583
Liabilities assumed in business combination	0	0	1,107,036
Effective settlement of subordinated notes in business combination	0	0	650
Change in goodwill	26,826	0	7,206
Cumulative effect adjustment due to adoption of accounting standard, net of income taxes	$ (8,111)	$ 0	$ 0
Cumulative effect adjustment of change in accounting method, net of income taxes	—	3,542	—