Fair Value - Carrying Amount and Estimated Fair Values of Financial Instruments (Details) - USD ($) $ in Thousands	Sep. 30, 2016	Dec. 31, 2015
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Accrued interest receivable	$ 4,165	$ 3,884
Accrued interest payable	446	461
Estimated fair value	28,838	18,607
Impaired collateral-dependent loans	2,878	6,331
Corporate and other securities
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Estimated fair value	13,231	1,971
Fair Value, Inputs, Level 3
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Impaired collateral-dependent loans	2,878	6,331
Carrying amount | Fair Value, Inputs, Level 1
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Cash and cash equivalents	84,670	88,157
Carrying amount | Fair Value, Inputs, Level 2
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities	72,360	71,336
SBA loans held for sale	15,611	13,114
Loans, net of allowance for loan losses	921,536	863,085
FHLB stock	5,767	4,600
Accrued interest receivable	4,165	3,884
Deposits	933,320	894,493
Borrowed funds and subordinated debentures	125,310	107,465
Accrued interest payable	446	461
Carrying amount | Fair Value, Inputs, Level 3
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Servicing assets	2,042	1,389
OREO	1,703	1,591
Carrying amount | Fair Value, Inputs, Level 3 | Corporate and other securities
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Estimated fair value	3,800	3,900
Estimated fair value | Fair Value, Inputs, Level 1
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Cash and cash equivalents	84,670	88,157
Estimated fair value | Fair Value, Inputs, Level 2
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Securities	73,024	71,472
SBA loans held for sale	17,407	14,324
Loans, net of allowance for loan losses	926,398	864,691
FHLB stock	5,767	4,600
Accrued interest receivable	4,165	3,884
Deposits	934,901	893,651
Borrowed funds and subordinated debentures	126,724	109,549
Accrued interest payable	446	461
Estimated fair value | Fair Value, Inputs, Level 3
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Servicing assets	2,042	1,389
OREO	1,703	1,591
Impaired collateral-dependent loans	2,900	6,300
Estimated fair value | Fair Value, Inputs, Level 3 | Corporate and other securities
Fair Value, Balance Sheet Grouping, Financial Statement Captions [Line Items]
Estimated fair value	$ 3,900	$ 3,800