Allowance for Loan Losses and Reserve for Unfunded Loan Commitments - Allowance for Credit Losses on Basis of Impairment Method (Details) - USD ($)	Mar. 31, 2017	Dec. 31, 2016	Mar. 31, 2016	Dec. 31, 2015
Financing Receivable, Allowance for Credit Losses [Line Items]
Total	$ 12,681,000	$ 12,579,000	$ 12,634,000	$ 12,759,000
Total loans	988,514,000	958,641,000
SBA loans held for investment
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	277,000	246,000
Collectively evaluated for impairment	1,376,000	1,330,000
Total	1,653,000	1,576,000	1,713,000	1,961,000
Individually evaluated for impairment	1,179,000	1,108,000
Collectively evaluated for impairment	41,224,000	41,384,000
Total loans	42,403,000	42,492,000
SBA 504 loans
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	0	0
Collectively evaluated for impairment	600,000	573,000
Total	600,000	573,000	598,000	741,000
Individually evaluated for impairment	237,000	513,000
Collectively evaluated for impairment	24,874,000	25,831,000
Total loans	25,111,000	26,344,000
Total commercial loans
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	19,000	34,000
Collectively evaluated for impairment	6,514,000	6,695,000
Total	6,533,000	6,729,000	6,311,000	6,309,000
Individually evaluated for impairment	1,499,000	426,000
Collectively evaluated for impairment	517,839,000	508,745,000
Total loans	519,338,000	509,171,000
Residential mortgage loans
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	0	0
Collectively evaluated for impairment	2,869,000	2,593,000
Total	2,869,000	2,593,000	2,892,000	2,769,000
Individually evaluated for impairment	0	0
Collectively evaluated for impairment	305,578,000	289,093,000
Total loans	305,578,000	289,093,000
Total consumer loans
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	0	0
Collectively evaluated for impairment	1,026,000	925,000
Total	1,026,000	925,000	836,000	817,000
Individually evaluated for impairment	0	0
Collectively evaluated for impairment	96,084,000	91,541,000
Total loans	96,084,000	91,541,000
Unallocated
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	0	0
Collectively evaluated for impairment	0	183,000
Total	0	183,000	$ 284,000	$ 162,000
Individually evaluated for impairment	0	0
Collectively evaluated for impairment	0	0
Total loans	0	0
Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	296,000	280,000
Collectively evaluated for impairment	12,385,000	12,299,000
Total	12,681,000	12,579,000
Individually evaluated for impairment	2,915,000	2,047,000
Collectively evaluated for impairment	985,599,000	956,594,000
Total loans	$ 988,514,000	$ 958,641,000