Fair Value - Carrying Amount and Estimated Fair Values of Financial Instruments (Details) - USD ($)	Sep. 30, 2018	Dec. 31, 2017
Financial assets:
Accrued interest receivable	$ 6,412,000	$ 5,447,000
Financial liabilities:
Accrued interest payable	506,000	436,000
Estimated fair value	15,016,000	16,346,000
Impaired collateral-dependent loans	2,765,000	1,126,000
Corporate and other securities
Financial liabilities:
Estimated fair value	4,579,000	4,614,000
Fair Value, Inputs, Level 1
Financial liabilities:
Impaired collateral-dependent loans	2,765,000	1,126,000
Fair Value, Inputs, Level 2
Financial liabilities:
Impaired collateral-dependent loans	0	0
Fair Value, Inputs, Level 3
Financial liabilities:
Impaired collateral-dependent loans	0	0
Carrying amount | Fair Value, Inputs, Level 1
Financial assets:
Cash and cash equivalents	144,556,000	150,254,000
Carrying amount | Fair Value, Inputs, Level 2
Financial assets:
Securities	63,399,000	69,800,000
SBA loans held for sale	13,029,000	22,810,000
Loans, net of allowance for loan losses	1,255,287,000	1,134,308,000
FHLB stock	9,445,000	12,863,000
Accrued interest receivable	6,412,000	5,447,000
Financial liabilities:
Deposits	1,219,473,000	1,043,137,000
Borrowed funds and subordinated debentures	190,310,000	285,310,000
Accrued interest payable	506,000	436,000
Carrying amount | Fair Value, Inputs, Level 3
Financial assets:
Servicing assets	2,135,000	1,800,000
OREO	56,000	426,000
Carrying amount | Fair Value, Inputs, Level 3 | Corporate and other securities
Financial liabilities:
Estimated fair value	3,600,000	3,700,000
Estimated fair value | Fair Value, Inputs, Level 1
Financial assets:
Cash and cash equivalents	144,556,000	150,254,000
Estimated fair value | Fair Value, Inputs, Level 2
Financial assets:
Securities	63,090,000	69,839,000
SBA loans held for sale	14,220,000	25,568,000
Loans, net of allowance for loan losses	1,234,657,000	1,133,739,000
FHLB stock	9,445,000	12,863,000
Accrued interest receivable	6,412,000	5,447,000
Financial liabilities:
Deposits	1,215,309,000	1,041,111,000
Borrowed funds and subordinated debentures	188,243,000	284,117,000
Accrued interest payable	506,000	436,000
Estimated fair value | Fair Value, Inputs, Level 3
Financial assets:
Servicing assets	2,135,000	1,800,000
OREO	56,000	426,000
Financial liabilities:
Impaired collateral-dependent loans	2,800	1,100,000
Estimated fair value | Fair Value, Inputs, Level 3 | Corporate and other securities
Financial liabilities:
Estimated fair value	$ 3,400,000	$ 3,500,000