Segment Reporting - Narrative (Details) $ in Thousands	3 Months Ended						9 Months Ended
	Sep. 30, 2020 USD ($)	Jun. 30, 2020 USD ($)	Mar. 31, 2020 USD ($)	Sep. 30, 2019 USD ($)	Jun. 30, 2019 USD ($)	Mar. 31, 2019 USD ($)	Sep. 30, 2020 USD ($) segment	Sep. 30, 2019 USD ($)
Segment Reporting Information [Line Items]
Number of reportable segments | segment							3
Net income (loss)	$ 6,491	$ 18,034	$ 1,048	$ 4,327	$ 15,377	$ 3,192	$ 25,573	$ 22,896
Repayments of long-term debt	10,800
Increase (decrease) in professional fees	1,606			1,316			5,931	3,025
Gain on sale of banking centers	0			0			9,631	0
Equity method investment income	13,620			0			13,620	0
Net income from continuing operations	6,491			4,346			25,573	22,469
Operating Segments | Commercial & Retail Banking
Segment Reporting Information [Line Items]
Net income (loss)	(3,700)			4,064			9,238	22,959
Increase (decrease) in net interest income	1,700						5,100
Increase (decrease) in loan interest and fee income	(421)						864
Increase (decrease) in deposit interest	$ (2,600)						$ (4,300)
Increase (decrease) in basis points on loan yield	(0.0056)						(0.0044)
Increase (decrease) in average balance of loans	$ 157,600						$ 164,900
Increase (decrease) in basis points	(0.0095)						(0.0072)
Increase (decrease) in average balance of deposits	$ 117,500						$ 129,600
Debt prepayment expense	500						500
Repayments of long-term debt	10,800
Increase (decrease) in noninterest income	847						4,700
Increase (Decrease) in consulting income	1,100						3,000
Increase (decrease) in commercial swap fee income	(616)
Increase (decrease) in noninterest expense	5,000						13,300
Increase (decrease) in salaries and employee benefits expense	2,700						6,100
Increase (decrease) In other operating expenses	802						1,500
Increase (decrease) in professional fees	572						3,100
Increase (decrease) in data processing expense	485						1,400
Increase (decrease) occupancy and equipment expense	277
Increase (Decrease) in insurance tax	176
Increase (decrease) in provision expense	8,000						14,900
Gain on sale of banking centers							9,600
Bargain purchase gain							4,700
Increase (decrease) in gain on sale of securities							1,000
Increase (decrease) in holding gain on equity securities							(13,500)
Increase (decrease) in travel, entertainment, dues, and subscriptions							579
Net income from continuing operations				4,064				22,959
Operating Segments | Commercial & Retail Banking | The First State Bank
Segment Reporting Information [Line Items]
Accretion expense							1,000
Operating Segments | Mortgage Banking
Segment Reporting Information [Line Items]
Net income (loss)	12,244			1,959			22,008	4,196
Increase (decrease) in net interest income	(631)						1,300
Increase (decrease) in average balance of loans	(145,500)						30,400
Increase (decrease) in noninterest income	4,100						22,300
Increase (decrease) in noninterest expense	(10,300)						(194)
Increase (decrease) in salaries and employee benefits expense	(8,200)						1,000
Increase (decrease) in professional fees							365
Increase (decrease) occupancy and equipment expense	(357)						(390)
Increase (decrease) in travel, entertainment, dues, and subscriptions	(456)						(436)
Increase (decrease) in interest and fees on loans	(2,200)						176
Equity method investment income	13,600
Gain on mortgage combination transaction	3,300						3,300
Increase (decrease) in derivative performance	(8,600)
Increase (decrease) in mortgage fee income	(4,300)						5,700
Increase (decrease) in mortgage closed loan volume							304,400
Increase (decrease) in mortgage processing expense	(724)						(509)
Increase (decrease) in FHLB and other borrowings interest income	(1,600)						(1,200)
Net income from continuing operations				1,959				4,196
Operating Segments | Financial Holding Company
Segment Reporting Information [Line Items]
Net income (loss)	(2,053)			(1,696)			(5,673)	(4,259)
Increase (decrease) in net interest income	(136)						(650)
Increase (decrease) in noninterest income	(35)						(126)
Increase (decrease) in noninterest expense	628						1,900
Increase (decrease) in salaries and employee benefits expense	611						1,700
Increase (decrease) in professional fees	140						533
Increase (decrease) in gain on sale of securities	17						(168)
Increase (decrease) in holding gain on equity securities	(69)
Net income from continuing operations	(2,100)			$ (1,677)			(5,700)	$ (4,686)
Increase (decrease) in income tax expense	(160)						(420)
Increase (decrease) in interest on subordinated debt	$ (136)						$ (650)