Segment Information (Details) (USD $) In Thousands, unless otherwise specified	3 Months Ended		6 Months Ended
	Jun. 30, 2014	Jun. 30, 2013	Jun. 30, 2014 Segment	Jun. 30, 2013
Segment Information [Abstract]
Number of business units			2
Revenue and income before income taxes for each business unit [Abstract]
Total revenue	$ 41,586	$ 44,648	$ 82,746	$ 81,298
Income before income taxes	986	6,063	1,979	7,074
Corporate and Other [Member]
Revenue and income before income taxes for each business unit [Abstract]
Total revenue	146	778	267	1,000
Income before income taxes	(1,286)	(801)	(2,998)	(2,269)
American Southern [Member] | Operating Segments [Member]
Revenue and income before income taxes for each business unit [Abstract]
Total revenue	14,147	14,747	28,348	25,237
Income before income taxes	1,138	3,177	2,195	4,619
Bankers Fidelity [Member] | Operating Segments [Member]
Revenue and income before income taxes for each business unit [Abstract]
Total revenue	27,293	29,123	54,131	55,061
Income before income taxes	$ 1,134	$ 3,687	$ 2,782	$ 4,724