Schedule II - Condensed Financial Information of Registrant (Parent Company Only) (Cash Flows) (Details) - USD ($) $ in Thousands	12 Months Ended
	Dec. 31, 2019	Dec. 31, 2018
Cash flows from operating activities:
Net (loss) income	$ (625)	$ 10,347
Adjustments to reconcile net (loss) income to cash provided by (used in) operating activities:
Depreciation and amortization expense	5,365	5,141
Deferred income tax (benefit) expense	817	(1,844)
Change in current federal income tax payable (recoverable)	(8,999)	7,536
Change in all other liabilities	5,158	(267)
Change in all other assets	4,273	3,007
Net cash provided by (used in) operating activities	27,670	(32,935)
Cash flows from investing activities:
Purchases of property and equipment	(4,188)	(2,101)
Purchase of investment securities	(259,769)	(222,642)
Maturities, sales and redemptions of investment securities	231,603	232,081
Net cash (used in) provided by investing activities	(32,354)	7,338
Cash flows from financing activities:
Proceeds from exercise of employee stock options	1,520	242
Payment of revolving credit facility	(30,000)
Payment of debt issuance costs	(979)
Proceeds from senior unsecured note offering	50,000
Purchase of treasury shares	(1,380)	(1,807)
Net cash provided by (used in) financing activities	19,161	(1,565)
Increase (decrease) in cash and cash equivalents and restricted cash	14,477	(27,162)
Cash and cash equivalents and restricted cash at beginning of period	40,471	67,633
Cash and cash equivalents and restricted cash at end of period	54,948	40,471
Supplemental cash flow information:
Interest paid	4,289	4,842
Income taxes (recovered) paid	7,775	(3,236)
Parent Company [Member]
Cash flows from operating activities:
Net (loss) income	(625)	10,347
Adjustments to reconcile net (loss) income to cash provided by (used in) operating activities:
Depreciation and amortization expense	381	377
Deferred income tax (benefit) expense	(611)	51
Net realized gains	(830)
Undistributed share of loss (earnings) of subsidiaries	14,183	(5,430)
Change in current federal income tax payable (recoverable)	229	(2,219)
Change in all other liabilities	994	(3,075)
Change in all other assets	(2,339)	(466)
Net cash provided by (used in) operating activities	11,382	(415)
Cash flows from investing activities:
Purchases of property and equipment	(1,211)	(55)
Purchase of investment securities	(1,259)
Maturities, sales and redemptions of investment securities	1,405
Capital contribution to subsidiaries	(20,000)
Net cash (used in) provided by investing activities	(21,065)	(55)
Cash flows from financing activities:
Proceeds from exercise of employee stock options	1,520	242
Payment of revolving credit facility	(30,000)
Payment of debt issuance costs	(979)
Proceeds from senior unsecured note offering	50,000
Purchase of treasury shares	(1,380)	(1,807)
Net cash provided by (used in) financing activities	19,161	(1,565)
Increase (decrease) in cash and cash equivalents and restricted cash	9,478	(2,035)
Cash and cash equivalents and restricted cash at beginning of period	10,159	12,194
Cash and cash equivalents and restricted cash at end of period	19,637	10,159
Supplemental cash flow information:
Interest paid	4,289	4,842
Income taxes (recovered) paid	$ (448)	$ 1,996