Fair Value - Carrying Amount and Estimated Fair Values of Financial Instruments (Details) - USD ($)	Jun. 30, 2018	Dec. 31, 2017
Financial assets:
Accrued interest receivable	$ 6,027,000	$ 5,447,000
Financial liabilities:
Accrued interest payable	509,000	436,000
Estimated fair value	15,614,000	16,346,000
Impaired collateral-dependent loans	3,534,000	1,126,000
Corporate and other securities
Financial liabilities:
Estimated fair value	4,597,000	4,614,000
Fair Value, Inputs, Level 1
Financial liabilities:
Impaired collateral-dependent loans	3,534,000	1,126,000
Fair Value, Inputs, Level 2
Financial liabilities:
Impaired collateral-dependent loans	0	0
Fair Value, Inputs, Level 3
Financial liabilities:
Impaired collateral-dependent loans	0	0
Carrying amount | Fair Value, Inputs, Level 1
Financial assets:
Cash and cash equivalents	139,275,000	150,254,000
Carrying amount | Fair Value, Inputs, Level 2
Financial assets:
Securities	65,682,000	69,800,000
SBA loans held for sale	14,889,000	22,810,000
Loans, net of allowance for loan losses	1,216,988,000	1,134,308,000
FHLB stock	11,245,000	12,863,000
Accrued interest receivable	6,027,000	5,447,000
Financial liabilities:
Deposits	1,146,399,000	1,043,137,000
Borrowed funds and subordinated debentures	230,310,000	285,310,000
Accrued interest payable	509,000	436,000
Carrying amount | Fair Value, Inputs, Level 3
Financial assets:
Servicing assets	1,985,000	1,800,000
OREO	56,000	426,000
Carrying amount | Fair Value, Inputs, Level 3 | Corporate and other securities
Financial liabilities:
Estimated fair value	3,600,000	3,700,000
Estimated fair value | Fair Value, Inputs, Level 1
Financial assets:
Cash and cash equivalents	139,275,000	150,254,000
Estimated fair value | Fair Value, Inputs, Level 2
Financial assets:
Securities	65,519,000	69,839,000
SBA loans held for sale	16,368,000	25,568,000
Loans, net of allowance for loan losses	1,208,298,000	1,133,739,000
FHLB stock	11,245,000	12,863,000
Accrued interest receivable	6,027,000	5,447,000
Financial liabilities:
Deposits	1,142,797,000	1,041,111,000
Borrowed funds and subordinated debentures	228,267,000	284,117,000
Accrued interest payable	509,000	436,000
Estimated fair value | Fair Value, Inputs, Level 3
Financial assets:
Servicing assets	1,985,000	1,800,000
OREO	56,000	426,000
Financial liabilities:
Impaired collateral-dependent loans	3,500	1,100,000
Estimated fair value | Fair Value, Inputs, Level 3 | Corporate and other securities
Financial liabilities:
Estimated fair value	$ 3,400,000	$ 3,500,000