Allowance for Loan Losses and Reserve for Unfunded Loan Commitments - Allowance for Credit Losses on Financing Receivables on Basis of Impairment Method (Details) - USD ($) $ in Thousands	Sep. 30, 2016	Jun. 30, 2016	Dec. 31, 2015	Sep. 30, 2015	Jun. 30, 2015	Dec. 31, 2014
Financing Receivable, Allowance for Credit Losses [Line Items]
Total	$ 12,685	$ 12,758	$ 12,759	$ 12,421	$ 12,404	$ 12,551
Total loans	934,221		875,844
SBA loans held for investment
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	351		705
Collectively evaluated for impairment	1,284		1,256
Total	1,635	1,630	1,961	1,911	1,998	1,883
Individually evaluated for impairment	1,343		1,907
Collectively evaluated for impairment	40,452		37,486
Total loans	41,795		39,393
SBA 504 loans
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	567		741
Total	567	588	741	790	879	1,337
Individually evaluated for impairment	513		2,226
Collectively evaluated for impairment	25,554		27,127
Total loans	26,067		29,353
Total commercial loans
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	253		137
Collectively evaluated for impairment	6,559		6,172
Total	6,812	6,566	6,309	6,200	6,059	6,270
Individually evaluated for impairment	1,626		3,040
Collectively evaluated for impairment	494,382		462,478
Total loans	496,008		465,518
Residential mortgage loans
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	2,516		2,769
Total	2,516	2,964	2,769	2,681	2,545	2,289
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	282,317		264,523
Total loans	282,317		264,523
Total consumer loans
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	892		817
Total	892	861	817	788	756	667
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	88,034		77,057
Total loans	88,034		77,057
Unallocated
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	263		162
Total	263	$ 149	162	$ 51	$ 167	$ 105
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	0		0
Total loans	0		0
Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	604		842
Collectively evaluated for impairment	12,081		11,917
Total	12,685		12,759
Individually evaluated for impairment	3,482		7,173
Collectively evaluated for impairment	930,739		868,671
Total loans	$ 934,221		$ 875,844