Segment Reporting (Details) - USD ($) $ in Thousands	3 Months Ended						9 Months Ended
	Sep. 30, 2020	Jun. 30, 2020	Mar. 31, 2020	Sep. 30, 2019	Jun. 30, 2019	Mar. 31, 2019	Sep. 30, 2020	Sep. 30, 2019	Dec. 31, 2019	Dec. 31, 2018
Segment Reporting Information [Line Items]
Interest income	$ 18,627			$ 21,038			$ 61,100	$ 61,131
Interest expense	2,617			6,004			10,461	17,596
NET INTEREST INCOME	16,010			15,034			50,639	43,535
Provision for loan losses	8,631			657			16,365	1,557
Net interest income after provision for loan losses	7,379			14,377			34,274	41,978
Noninterest Income:
Mortgage fee income	7,264			11,496			33,427	28,030
Other income	12,134			3,200			42,334	21,818
Total noninterest income	19,398			14,696			75,761	49,848
Noninterest Expenses:
Salaries and employee benefits	10,519			15,438			49,360	40,452
Other expense	7,746			7,942			26,894	21,766
Total noninterest expenses	18,265			23,380			76,254	62,218
Income from continuing operations, before income taxes	8,512			5,693			33,781	29,608
Income tax expense (benefit)	2,021			1,347			8,208	7,139
Net income from continuing operations	6,491			4,346			25,573	22,469
Income (loss) from discontinued operations, before income taxes	0			(25)			0	575
Income tax expense (benefit) - discontinued operations	0			(6)			0	148
Net income (loss) from discontinued operations	0			(19)			0	427
Net income	6,491	$ 18,034	$ 1,048	4,327	$ 15,377	$ 3,192	25,573	22,896
Preferred dividends	116			121			345	364
Net income available to common shareholders - basic	6,375			4,206			25,228	22,532
Capital expenditures	1,652			514			4,211	1,134
Assets	2,214,459			1,961,952			2,214,459	1,961,952	$ 1,944,114	$ 1,750,969
Goodwill	2,350			19,630			2,350	19,630	19,630	18,480
Operating Segments | Commercial & Retail Banking
Segment Reporting Information [Line Items]
Interest income	18,737			19,299			56,693	56,446
Interest expense	2,553			4,806			9,418	14,303
NET INTEREST INCOME	16,184			14,493			47,275	42,143
Provision for loan losses	8,631			625			16,361	1,497
Net interest income after provision for loan losses	7,553			13,868			30,914	40,646
Noninterest Income:
Mortgage fee income	26			121			176	507
Other income	3,080			2,138			24,218	19,168
Total noninterest income	3,106			2,259			24,394	19,675
Noninterest Expenses:
Salaries and employee benefits	7,526			4,820			19,562	13,435
Other expense	8,389			6,113			24,172	16,958
Total noninterest expenses	15,915			10,933			43,734	30,393
Income from continuing operations, before income taxes	(5,256)			5,194			11,574	29,928
Income tax expense (benefit)	(1,556)			1,130			2,336	6,969
Net income from continuing operations				4,064				22,959
Income (loss) from discontinued operations, before income taxes				0				0
Income tax expense (benefit) - discontinued operations				0				0
Net income (loss) from discontinued operations				0				0
Net income	(3,700)			4,064			9,238	22,959
Preferred dividends	0			0			0	0
Net income available to common shareholders - basic	(3,700)			4,064			9,238	22,959
Capital expenditures	1,652			414			4,092	915
Assets	2,225,615			1,959,817			2,225,615	1,959,817	1,953,975	1,753,932
Goodwill	2,350			2,748			2,350	2,748	2,748	1,598
Operating Segments | Mortgage Banking
Segment Reporting Information [Line Items]
Interest income	78			2,288			6,034	5,858
Interest expense	232			1,811			3,136	4,303
NET INTEREST INCOME	(154)			477			2,898	1,555
Provision for loan losses	0			32			4	60
Net interest income after provision for loan losses	(154)			445			2,894	1,495
Noninterest Income:
Mortgage fee income	7,238			11,587			33,793	28,076
Other income	9,555			1,112			19,347	2,723
Total noninterest income	16,793			12,699			53,140	30,799
Noninterest Expenses:
Salaries and employee benefits	82			8,318			21,550	20,515
Other expense	68			2,142			4,780	6,009
Total noninterest expenses	150			10,460			26,330	26,524
Income from continuing operations, before income taxes	16,489			2,684			29,704	5,770
Income tax expense (benefit)	4,245			725			7,696	1,574
Net income from continuing operations				1,959				4,196
Income (loss) from discontinued operations, before income taxes				0				0
Income tax expense (benefit) - discontinued operations				0				0
Net income (loss) from discontinued operations				0				0
Net income	12,244			1,959			22,008	4,196
Preferred dividends	0			0			0	0
Net income available to common shareholders - basic	12,244			1,959			22,008	4,196
Capital expenditures	0			23			99	83
Assets	56,646			272,007			56,646	272,007	248,382	165,430
Goodwill	0			16,882			0	16,882	16,882	16,882
Operating Segments | Financial Holding Company
Segment Reporting Information [Line Items]
Interest income	0			9			2	12
Interest expense	20			156			78	728
NET INTEREST INCOME	(20)			(147)			(76)	(716)
Provision for loan losses	0			0			0	0
Net interest income after provision for loan losses	(20)			(147)			(76)	(716)
Noninterest Income:
Mortgage fee income	0			0			0	0
Other income	1,481			1,516			4,664	4,790
Total noninterest income	1,481			1,516			4,664	4,790
Noninterest Expenses:
Salaries and employee benefits	2,911			2,300			8,248	6,502
Other expense	1,271			1,254			3,837	3,662
Total noninterest expenses	4,182			3,554			12,085	10,164
Income from continuing operations, before income taxes	(2,721)			(2,185)			(7,497)	(6,090)
Income tax expense (benefit)	(668)			(508)			(1,824)	(1,404)
Net income from continuing operations	(2,100)			(1,677)			(5,700)	(4,686)
Income (loss) from discontinued operations, before income taxes				(25)				575
Income tax expense (benefit) - discontinued operations				(6)				148
Net income (loss) from discontinued operations				(19)				427
Net income	(2,053)			(1,696)			(5,673)	(4,259)
Preferred dividends	116			121			345	364
Net income available to common shareholders - basic	(2,169)			(1,817)			(6,018)	(4,623)
Capital expenditures	0			77			20	136
Assets	238,564			210,715			238,564	210,715	216,411	196,537
Goodwill	0			0			0	0	0	0
Intercompany Eliminations
Segment Reporting Information [Line Items]
Interest income	(188)			(558)			(1,629)	(1,185)
Interest expense	(188)			(769)			(2,171)	(1,738)
NET INTEREST INCOME	0			211			542	553
Provision for loan losses	0			0			0	0
Net interest income after provision for loan losses	0			211			542	553
Noninterest Income:
Mortgage fee income	0			(212)			(542)	(553)
Other income	(1,982)			(1,566)			(5,895)	(4,863)
Total noninterest income	(1,982)			(1,778)			(6,437)	(5,416)
Noninterest Expenses:
Salaries and employee benefits	0			0			0	0
Other expense	(1,982)			(1,567)			(5,895)	(4,863)
Total noninterest expenses	(1,982)			(1,567)			(5,895)	(4,863)
Income from continuing operations, before income taxes	0			0			0	0
Income tax expense (benefit)	0			0			0	0
Net income from continuing operations				0				0
Income (loss) from discontinued operations, before income taxes				0				0
Income tax expense (benefit) - discontinued operations				0				0
Net income (loss) from discontinued operations				0				0
Net income	0			0			0	0
Preferred dividends	0			0			0	0
Net income available to common shareholders - basic	0			0			0	0
Capital expenditures	0			0			0	0
Assets	(306,366)			(480,587)			(306,366)	(480,587)	(474,654)	(364,930)
Goodwill	$ 0			$ 0			$ 0	$ 0	$ 0	$ 0