Loans and Asset Quality Information - Summary of Nonperforming Assets (Details) - USD ($) $ in Thousands	Dec. 31, 2020	Dec. 31, 2019	Dec. 31, 2018	Oct. 01, 2017	Mar. 03, 2017
Loans and Leases Receivable Disclosure [Line Items]
Nonaccrual loans	$ 35,076	$ 24,866
Total loans	4,735,013	4,451,525	$ 4,247,765
Foreclosed properties	2,424	3,873
Purchased credit impaired loans not included above	8,591	12,664
Nonperforming assets
Loans and Leases Receivable Disclosure [Line Items]
Nonaccrual loans	35,076	24,866
TDRs- accruing	9,497	9,053
Accruing loans 90 days past due	0	0
Total loans	44,573	33,919
Foreclosed properties	2,424	3,873
Total nonperforming assets	46,997	37,792
Carolina Bank | Purchased Impaired Loans
Loans and Leases Receivable Disclosure [Line Items]
Recorded loans with a fair value					$ 19,300
Asheville Savings Bank | Purchased Impaired Loans
Loans and Leases Receivable Disclosure [Line Items]
Recorded loans with a fair value				$ 9,900
Carolina Bank and Asheville Savings Bank | Accruing 90 Days or More Past Due | Purchased Impaired Loans
Loans and Leases Receivable Disclosure [Line Items]
Recorded loans with a fair value	$ 700	$ 800