Segment Information (Details) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2020 USD ($)	Jun. 30, 2019 USD ($)	Jun. 30, 2020 USD ($) Segment	Jun. 30, 2019 USD ($)	Dec. 31, 2019 USD ($)
Segment Information [Abstract]
Number of business units | Segment			2
Assets, Revenue and Income (loss) before income taxes for each business unit [Abstract]
Assets	$ 385,737		$ 385,737		$ 377,626
Revenue	49,737	$ 43,127	89,147	$ 98,145
Income (loss) before income taxes	8,281	(5,589)	(1,946)	(304)
Operating Segments [Member] | American Southern [Member]
Assets, Revenue and Income (loss) before income taxes for each business unit [Abstract]
Assets	153,632		153,632		141,524
Revenue	16,896	15,740	32,123	30,975
Income (loss) before income taxes	2,087	1,396	2,965	3,378
Operating Segments [Member] | Bankers Fidelity [Member]
Assets, Revenue and Income (loss) before income taxes for each business unit [Abstract]
Assets	220,111		220,111		224,122
Revenue	32,871	31,244	57,744	65,620
Income (loss) before income taxes	8,039	(1,998)	(742)	(2,494)
Corporate and Other [Member]
Assets, Revenue and Income (loss) before income taxes for each business unit [Abstract]
Assets	11,994		11,994		$ 11,980
Revenue	(30)	(3,857)	(720)	1,550
Income (loss) before income taxes	$ (1,845)	$ (4,987)	$ (4,169)	$ (1,188)