Allowance for Loan Losses and Reserve for Unfunded Loan Commitments - Allowance for Credit Losses on Financing Receivables on Basis of Impairment Method (Details) - USD ($)	Jun. 30, 2017	Mar. 31, 2017	Dec. 31, 2016	Jun. 30, 2016	Mar. 31, 2016	Dec. 31, 2015
Financing Receivable, Allowance for Credit Losses [Line Items]
Total	$ 12,800,000	$ 12,681,000	$ 12,579,000	$ 12,758,000	$ 12,634,000	$ 12,759,000
Total loans	1,032,854,000		958,641,000
SBA loans held for investment
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	149,000		246,000
Collectively evaluated for impairment	1,365,000		1,330,000
Total	1,514,000	1,653,000	1,576,000	1,630,000	1,713,000	1,961,000
Individually evaluated for impairment	848,000		1,108,000
Collectively evaluated for impairment	42,481,000		41,384,000
Total loans	43,329,000		42,492,000
SBA 504 loans
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	494,000		573,000
Total	494,000	600,000	573,000	588,000	598,000	741,000
Individually evaluated for impairment	0		513,000
Collectively evaluated for impairment	23,153,000		25,831,000
Total loans	23,153,000		26,344,000
Commercial loans
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	19,000		34,000
Collectively evaluated for impairment	6,874,000		6,695,000
Total	6,893,000	6,533,000	6,729,000	6,566,000	6,312,000	6,309,000
Individually evaluated for impairment	411,000		426,000
Collectively evaluated for impairment	544,897,000		508,745,000
Total loans	545,308,000		509,171,000
Residential mortgage loans
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	2,816,000		2,593,000
Total	2,816,000	2,869,000	2,593,000	2,964,000	2,892,000	2,769,000
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	315,396,000		289,093,000
Total loans	315,396,000		289,093,000
Consumer loans
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	1,083,000		925,000
Total	1,083,000	1,026,000	925,000	861,000	836,000	817,000
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	105,668,000		91,541,000
Total loans	105,668,000		91,541,000
Unallocated
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	0		183,000
Total	0	$ 0	183,000	$ 149,000	$ 283,000	$ 162,000
Individually evaluated for impairment	0		0
Collectively evaluated for impairment	0		0
Total loans	0		0
Total
Financing Receivable, Allowance for Credit Losses [Line Items]
Individually evaluated for impairment	168,000		280,000
Collectively evaluated for impairment	12,632,000		12,299,000
Total	12,800,000		12,579,000
Individually evaluated for impairment	1,259,000		2,047,000
Collectively evaluated for impairment	1,031,595,000		956,594,000
Total loans	$ 1,032,854,000		$ 958,641,000