Consolidated Statements of Operations - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Interest income:
Interest and fees on taxable loans	$ 72,808	$ 62,502	$ 48,993
Interest and fees on tax-exempt loans	1,689	1,543	1,101
Interest and dividends on taxable investment securities	27,459	26,677	19,643
Interest and dividends on tax-exempt investment securities	337	505	535
Interest on federal funds sold and other interest-earning assets	2,571	847	577
Total interest income	104,864	92,074	70,849
Interest expense:
Demand- interest bearing	15,621	7,946	3,020
Money market and savings	6,796	4,898	3,160
Time deposits	3,850	1,588	1,238
Other borrowings	790	1,738	1,366
Total interest expense	27,057	16,170	8,784
Net interest income	77,807	75,904	62,065
Provision for loan losses	1,905	2,300	900
Net interest income after provision for loan losses	75,902	73,604	61,165
Non-interest income:
Non-interest income	7,755	5,650	4,081
Gain on sales of SBA loans	3,187	3,105	3,378
Gain (loss) on sale of investment securities	1,103	(67)	(146)
Other non-interest income	214	174	177
Total non-interest income	23,738	20,322	20,097
Non-interest expenses:
Salaries and employee benefits	53,888	44,082	37,959
Occupancy	11,565	8,046	7,156
Depreciation and amortization	6,482	5,447	4,618
Legal	1,335	985	984
Other real estate owned	2,109	1,588	4,092
Appraisal and other loan expenses	1,829	1,840	1,878
Advertising	1,930	1,211	1,279
Data processing	5,220	3,855	3,134
Insurance	1,070	996	982
Professional fees	2,589	2,048	1,893
Debit card processing	2,467	1,868	1,264
Regulatory assessments and costs	1,228	1,675	1,367
Taxes, other	837	796	817
Other operating expenses	11,941	9,284	7,853
Total non-interest expense	104,490	83,721	75,276
Income (loss) before benefit for income taxes	(4,850)	10,205	5,986
Provision (benefit) for income taxes	(1,350)	1,578	(2,919)
Net income (loss)	$ (3,500)	$ 8,627	$ 8,905
Net income (loss) per share
Basic (in dollars per share)	$ (0.06)	$ 0.15	$ 0.16
Diluted (in dollars per share)	$ (0.06)	$ 0.15	$ 0.15
Loan and Servicing Fees [Member]
Non-interest income:
Non-interest income	$ 1,568	$ 1,401	$ 1,614
Mortgage Banking [Member]
Non-interest income:
Non-interest income	10,125	10,233	11,170
Deposit Account [Member]
Non-interest income:
Non-interest income	$ 7,541	$ 5,476	$ 3,904