Investment Securities (Details) $ in Thousands	3 Months Ended		9 Months Ended
	Sep. 30, 2016 USD ($) fund item	Sep. 30, 2015 USD ($)	Sep. 30, 2016 USD ($) item	Sep. 30, 2015 USD ($)	Dec. 31, 2015 USD ($) item
Carrying value and fair value of available for sale investment securities [Abstract]
Total available for sale securities	$ 165,787		$ 165,787		$ 224,108
Gross unrealized gains	3,058		3,058		2,205
Gross unrealized losses	(524)		(524)		(2,246)
Available-for-sale Securities, Total	168,321		168,321		224,067
Amortized cost [Abstract]
Within 1 year	0		0
After 1 but within 5 years	7,802		7,802
After 5 but within 10 years	5,102		5,102
After 10 years	1,000		1,000
Total available for sale debt securities	163,928		163,928
Available-for-sale Equity Securities, Amortized Cost Basis	1,859		1,859
Total available for sale securities	165,787		165,787		224,108
Fair value [Abstract]
Within 1 year	0		0
After 1 but within 5 years	7,798		7,798
After 5 but within 10 years	5,269		5,269
After 10 years	1,074		1,074
Total available for sale debt securities	166,462		166,462
No contractual maturity	1,859		1,859
Fair value	168,321		168,321		224,067
Proceeds from sales of AFS investment securities	25,700	$ 9,200	36,427	$ 33,653
Gross realized gains and losses on the sale of securities recognized in earnings [Abstract]
Gross realized gains	280	104	1,426	676
Gross realized losses	0	(24)	(44)	(24)
Net realized gains	280	80	1,382	652
Other than Temporary Impairment Losses, Investments, Portion Recognized in Earnings, Net, Available-for-sale Securities	35	$ 0	181	0
Available for sale securities, continuous unrealized loss position [Abstract]
Less than 12 months, Fair value	24,450		24,450		106,022
12 months or longer, Fair value	11,782		11,782		37,100
Total, Fair value	36,232		36,232		143,122
Less than 12 months, Gross unrealized losses	(273)		(273)		(1,134)
12 months or longer, Gross unrealized losses	(251)		(251)		(1,112)
Total, Gross unrealized losses	$ (524)		$ (524)		$ (2,246)
Less than 12 months, number of positions | item	10		10		43
12 months or longer, number of positions | item	5		5		12
Total , number of positions | item	15		15		55
Fair value	$ 168,321		$ 168,321		$ 224,067
Number of securities in an unrealized loss position for less than twelve months | item	10		10		43
Number of securities in an unrealized loss position for more than twelve months | item	5		5		12
Credit related impairment losses on debt securities held for which a portion of OTTI was recognized in other comprehensive income [Roll Forward]
Reductions for securities sold during the period (realized)			$ 0	$ 0
U.S. Government Agencies [Member]
Carrying value and fair value of available for sale investment securities [Abstract]
Total available for sale securities	$ 3,498		3,498		$ 26,127
Gross unrealized gains	2		2		0
Gross unrealized losses	0		0		(564)
Available-for-sale Securities, Total	3,500		3,500		25,563
Amortized cost [Abstract]
Total available for sale securities	3,498		3,498		26,127
Fair value [Abstract]
Fair value	3,500		3,500		25,563
Available for sale securities, continuous unrealized loss position [Abstract]
Less than 12 months, Fair value					6,681
12 months or longer, Fair value					18,882
Total, Fair value					25,563
Less than 12 months, Gross unrealized losses					(57)
12 months or longer, Gross unrealized losses					(507)
Total, Gross unrealized losses					$ (564)
Less than 12 months, number of positions | item					2
12 months or longer, number of positions | item					6
Total , number of positions | item					8
Fair value	3,500		3,500		$ 25,563
Number of securities in an unrealized loss position for less than twelve months | item					2
Number of securities in an unrealized loss position for more than twelve months | item					6
Mortgage-backed Securities-residential [Member]
Carrying value and fair value of available for sale investment securities [Abstract]
Total available for sale securities	7,373		7,373		$ 11,002
Gross unrealized gains	342		342		106
Gross unrealized losses	0		0		(50)
Available-for-sale Securities, Total	7,715		7,715		11,058
Amortized cost [Abstract]
Available for sale securities Debt maturities, amortized cost basis	7,373		7,373
Total available for sale securities	7,373		7,373		11,002
Fair value [Abstract]
Available-for-sale Securities, Debt Maturities, without Single Maturity Date, Fair Value	7,715		7,715
Fair value	7,715		7,715		11,058
Available for sale securities, continuous unrealized loss position [Abstract]
Less than 12 months, Fair value					5,140
12 months or longer, Fair value					2,574
Total, Fair value					7,714
Less than 12 months, Gross unrealized losses					(5)
12 months or longer, Gross unrealized losses					(45)
Total, Gross unrealized losses					$ (50)
Less than 12 months, number of positions | item					2
12 months or longer, number of positions | item					1
Total , number of positions | item					3
Fair value	7,715		7,715		$ 11,058
Number of securities in an unrealized loss position for less than twelve months | item					2
Number of securities in an unrealized loss position for more than twelve months | item					1
Collateralized Mortgage Obligations, Issued or Guaranteed by U.S. Government Agencies [Member]
Carrying value and fair value of available for sale investment securities [Abstract]
Total available for sale securities	138,137		138,137		$ 170,764
Gross unrealized gains	2,372		2,372		1,524
Gross unrealized losses	(469)		(469)		(1,554)
Available-for-sale Securities, Total	140,040		140,040		170,734
Amortized cost [Abstract]
Available for sale securities Debt maturities, amortized cost basis	138,137		138,137
Total available for sale securities	138,137		138,137		170,764
Fair value [Abstract]
Available-for-sale Securities, Debt Maturities, without Single Maturity Date, Fair Value	140,040		140,040
Fair value	140,040		140,040		170,734
Available for sale securities, continuous unrealized loss position [Abstract]
Less than 12 months, Fair value	22,424		22,424		87,020
12 months or longer, Fair value	10,458		10,458		15,644
Total, Fair value	32,882		32,882		102,664
Less than 12 months, Gross unrealized losses	(224)		(224)		(994)
12 months or longer, Gross unrealized losses	(245)		(245)		(560)
Total, Gross unrealized losses	$ (469)		$ (469)		$ (1,554)
Less than 12 months, number of positions | item	8		8		31
12 months or longer, number of positions | item	3		3		5
Total , number of positions | item	11		11		36
Fair value	$ 140,040		$ 140,040		$ 170,734
Number of securities in an unrealized loss position for less than twelve months | item	8		8		31
Number of securities in an unrealized loss position for more than twelve months | item	3		3		5
Mortgage-backed Securities, Issued by Private Enterprises [Member]
Carrying value and fair value of available for sale investment securities [Abstract]
Total available for sale securities	$ 4,514		$ 4,514		$ 2,729
Gross unrealized gains	52		52		1
Gross unrealized losses	0		0		(26)
Available-for-sale Securities, Total	4,566		4,566		2,704
Amortized cost [Abstract]
Available for sale securities Debt maturities, amortized cost basis	4,514		4,514
Total available for sale securities	4,514		4,514		2,729
Fair value [Abstract]
Available-for-sale Securities, Debt Maturities, without Single Maturity Date, Fair Value	4,566		4,566
Fair value	4,566		4,566		2,704
Available for sale securities, continuous unrealized loss position [Abstract]
Less than 12 months, Fair value					1,101
12 months or longer, Fair value					0
Total, Fair value					1,101
Less than 12 months, Gross unrealized losses					(26)
12 months or longer, Gross unrealized losses					0
Total, Gross unrealized losses					$ (26)
Less than 12 months, number of positions | item					1
12 months or longer, number of positions | item					0
Total , number of positions | item					1
Fair value	4,566		4,566		$ 2,704
Number of securities in an unrealized loss position for less than twelve months | item					1
Number of securities in an unrealized loss position for more than twelve months | item					0
Corporate Bonds [Member]
Carrying value and fair value of available for sale investment securities [Abstract]
Total available for sale securities	1,700		1,700		$ 1,500
Gross unrealized gains	74		74		56
Gross unrealized losses	0		0		0
Available-for-sale Securities, Total	1,774		1,774		1,556
Amortized cost [Abstract]
Total available for sale securities	1,700		1,700		1,500
Fair value [Abstract]
Fair value	1,774		1,774		1,556
Available for sale securities, continuous unrealized loss position [Abstract]
Fair value	1,774		1,774		1,556
Municipal Bonds [Member]
Carrying value and fair value of available for sale investment securities [Abstract]
Total available for sale securities	8,706		8,706		9,910
Gross unrealized gains	216		216		73
Gross unrealized losses	(55)		(55)		(52)
Available-for-sale Securities, Total	8,867		8,867		9,931
Amortized cost [Abstract]
Total available for sale securities	8,706		8,706		9,910
Fair value [Abstract]
Fair value	8,867		8,867		9,931
Available for sale securities, continuous unrealized loss position [Abstract]
Less than 12 months, Fair value	2,026		2,026		6,080
12 months or longer, Fair value	1,324		1,324		0
Total, Fair value	3,350		3,350		6,080
Less than 12 months, Gross unrealized losses	(49)		(49)		(52)
12 months or longer, Gross unrealized losses	(6)		(6)		0
Total, Gross unrealized losses	$ (55)		$ (55)		$ (52)
Less than 12 months, number of positions | item	2		2		7
12 months or longer, number of positions | item	2		2		0
Total , number of positions | item	4		4		7
Fair value	$ 8,867		$ 8,867		$ 9,931
Number of securities in an unrealized loss position for less than twelve months | item	2		2		7
Number of securities in an unrealized loss position for more than twelve months | item	2		2		0
Private Equity Funds [Member]
Carrying value and fair value of available for sale investment securities [Abstract]
Total available for sale securities	$ 1,833		$ 1,833		$ 2,050
Gross unrealized gains	0		0		445
Gross unrealized losses	0		0		0
Available-for-sale Securities, Total	1,833		1,833		2,495
Amortized cost [Abstract]
Total available for sale securities	1,833		1,833		2,050
Fair value [Abstract]
Fair value	1,833		1,833		2,495
Available for sale securities, continuous unrealized loss position [Abstract]
Fair value	$ 1,833		1,833		2,495
Impairment charges on private equity funds			181
Number of private equity funds that had impairment charges recorded on | fund	2
Common Stock [Member]
Carrying value and fair value of available for sale investment securities [Abstract]
Total available for sale securities	$ 26		26		26
Gross unrealized gains	0		0		0
Gross unrealized losses	0		0		0
Available-for-sale Securities, Total	26		26		26
Amortized cost [Abstract]
Total available for sale securities	26		26		26
Fair value [Abstract]
Fair value	26		26		26
Available for sale securities, continuous unrealized loss position [Abstract]
Fair value	$ 26		$ 26		$ 26