Allowance for Loan Losses and Reserve for Unfunded Loan Commitments - Allowance for Credit Losses on Financing Receivables on Basis of Impairment Method (Details) - USD ($) $ in Thousands	Jun. 30, 2016	Mar. 31, 2016	Dec. 31, 2015	Jun. 30, 2015	Mar. 31, 2015	Dec. 31, 2014
Financing Receivable, Allowance for Credit Losses [Line Items]
Total	$ 12,758	$ 12,634	$ 12,759	$ 12,404	$ 12,181	$ 12,551
Total loans	901,798		875,844
SBA loans held for investment
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	415		705
Collectively evaluated for impairment	1,215		1,256
Total	1,630	1,713	1,961	1,998	1,873	1,883
Individually evaluated for impairment	1,517		1,907
Collectively evaluated for impairment	38,489		37,486
Total loans	40,006		39,393
SBA 504 loans
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	588		741
Total	588	598	741	879	1,162	1,337
Individually evaluated for impairment	513		2,226
Collectively evaluated for impairment	26,525		27,127
Total loans	27,038		29,353
Total commercial loans
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	274		137
Collectively evaluated for impairment	6,292		6,172
Total	6,566	6,312	6,309	6,059	5,624	6,270
Individually evaluated for impairment	3,061		3,040
Collectively evaluated for impairment	478,652		462,478
Total loans	481,713		465,518
Residential mortgage loans
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	2,964		2,769
Total	2,964	2,892	2,769	2,545	2,341	2,289
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	268,774		264,523
Total loans	268,774		264,523
Total consumer loans
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	861		817
Total	861	836	817	756	710	667
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	84,267		77,057
Total loans	84,267		77,057
Unallocated
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	149		162
Total	149	$ 283	162	$ 167	$ 471	$ 105
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	0		0
Total loans	0		0
Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	689		842
Collectively evaluated for impairment	12,069		11,917
Total	12,758		12,759
Individually evaluated for impairment	5,091		7,173
Collectively evaluated for impairment	896,707		868,671
Total loans	$ 901,798		$ 875,844