Investment Securities - Summary of Gross Unrealized Losses and Fair Value (Details) - USD ($) $ in Thousands	Dec. 31, 2017	Dec. 31, 2016
Available-for-sale Securities, Continuous Unrealized Loss Position [Abstract]
Fair Value, 12 months or less	$ 30,145	$ 58,476
Unrealized Losses, 12 months or less	(237)	(994)
Fair Value, more than 12 months	23,772	2,624
Unrealized Losses, more than 12 months	(441)	(30)
Held-to-maturity Securities, Continuous Unrealized Loss Position [Abstract]
Fair Value, 12 months or less	0	9,601
Unrealized Losses, 12 months or less	0	(188)
Fair Value, more than 12 months	7,652	930
Unrealized Losses, more than 12 months	(156)	(21)
Obligations of U.S. Government sponsored enterprises
Available-for-sale Securities, Continuous Unrealized Loss Position [Abstract]
Fair Value, 12 months or less	14,974	10,050
Unrealized Losses, 12 months or less	(98)	(40)
Fair Value, more than 12 months	0	0
Unrealized Losses, more than 12 months	0	0
Municipal securities
Available-for-sale Securities, Continuous Unrealized Loss Position [Abstract]
Fair Value, 12 months or less	0	3,380
Unrealized Losses, 12 months or less	0	(54)
Fair Value, more than 12 months	1,050	0
Unrealized Losses, more than 12 months	(23)	0
Held-to-maturity Securities, Continuous Unrealized Loss Position [Abstract]
Fair Value, 12 months or less		1,536
Unrealized Losses, 12 months or less		(52)
Fair Value, more than 12 months		0
Unrealized Losses, more than 12 months		0
SBA pool securities
Available-for-sale Securities, Continuous Unrealized Loss Position [Abstract]
Fair Value, 12 months or less	3,285	13,912
Unrealized Losses, 12 months or less	(42)	(230)
Fair Value, more than 12 months	4,979	0
Unrealized Losses, more than 12 months	(85)	0
Residential mortgage-backed securities issued by GSE's
Available-for-sale Securities, Continuous Unrealized Loss Position [Abstract]
Fair Value, 12 months or less	1,835	7,539
Unrealized Losses, 12 months or less	(8)	(65)
Fair Value, more than 12 months	5,383	2,624
Unrealized Losses, more than 12 months	(68)	(30)
Held-to-maturity Securities, Continuous Unrealized Loss Position [Abstract]
Fair Value, 12 months or less	0	8,065
Unrealized Losses, 12 months or less	0	(136)
Fair Value, more than 12 months	7,652	930
Unrealized Losses, more than 12 months	(156)	(21)
Commercial mortgage-backed securities
Available-for-sale Securities, Continuous Unrealized Loss Position [Abstract]
Fair Value, 12 months or less	10,051	23,595
Unrealized Losses, 12 months or less	(89)	(605)
Fair Value, more than 12 months	12,360	0
Unrealized Losses, more than 12 months	$ (265)	$ 0