Parent Company Only Financial Information - (As Restated) (Condensed Statement of Financial Condition) (Details) - USD ($) $ in Thousands	Dec. 31, 2015	Dec. 31, 2014	Dec. 31, 2013
Cash	$ 52,141	$ 35,451	$ 43,063
Investment in bank subsidiary	1,300
Other Assets	19,495	21,853
Assets	1,323,458	1,332,296
Shareholders' equity	120,771	108,999	101,883
Liabilities and Stockholders' Equity	1,323,458	1,332,296
Parent Company [Member]
Cash	11,102	1,036	$ 3,025
Investment securities	14,522	0
Investment in bank subsidiary	132,809	150,209
Investment in non-bank subsidiaries	1,255	1,255
Other Assets	5,403	5,872
Assets	165,091	158,372
Accrued Liabilities and Other Liabilities	2,590	2,643
Dividends Payable	0	0
Junior Subordinated Notes	41,730	46,730
Shareholders' equity	120,771	108,999
Liabilities and Stockholders' Equity	$ 165,091	$ 158,372