Fair Value - Carrying Amount and Fair Values (Details) - USD ($) $ in Thousands	Jun. 30, 2020	Dec. 31, 2019
Carrying amount | Fair Value, Recurring [Member] | Corporate and other securities
Financial assets:
Securities	$ 6,200
Carrying amount | Quoted Prices in Active Markets for Identical Assets (Level 1)
Financial assets:
Cash and cash equivalents	186,683	$ 158,016
Carrying amount | Significant Other Observable Inputs (Level 2)
Financial assets:
Securities	54,888	66,564
SBA loans held for sale	10,602	13,529
Loans, net of allowance for loan losses	1,561,671	1,395,634
FHLB stock	11,629	14,184
Accrued interest receivable	11,039	6,984
Financial liabilities:
Deposits	1,483,457	1,250,114
Borrowed funds and subordinated debentures	233,310	293,310
Accrued interest payable	228	455
Carrying amount | Fair Value, Inputs, Level 3
Financial assets:
Servicing assets	2,138	2,026
OREO	711	1,723
Estimated fair value | Quoted Prices in Active Markets for Identical Assets (Level 1)
Financial assets:
Cash and cash equivalents	186,683	158,016
Estimated fair value | Significant Other Observable Inputs (Level 2)
Financial assets:
Securities	54,888	66,564
SBA loans held for sale	11,707	14,862
Loans, net of allowance for loan losses	1,581,341	1,398,997
FHLB stock	11,629	14,184
Accrued interest receivable	11,039	6,984
Financial liabilities:
Deposits	1,489,973	1,252,082
Borrowed funds and subordinated debentures	235,645	292,766
Accrued interest payable	228	455
Estimated fair value | Fair Value, Inputs, Level 3
Financial assets:
Servicing assets	2,138	2,026
OREO	711	$ 1,723
Estimated fair value | Fair Value, Inputs, Level 3 | Fair Value, Recurring [Member]
Financial assets:
Fair value on a recurring basis	$ 6,200