Note 21 - Fair Value and Interest Rate Risk - Financial Assets and Liabilities (Details) - USD ($)	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Fair value, total	$ 48,317,000	$ 39,496,000
SBA servicing assets	201,000		$ 0
Interest rate swap receivable	694,000	286,000
Reported Value Measurement [Member]
Financial assets, total	921,374,000	895,546,000
Financial liabilities, total	903,186,000	876,862,000
Estimate of Fair Value Measurement [Member]
Financial assets, total	914,414,000	885,360,000
Financial liabilities, total	907,534,000	875,279,000
Fair Value, Inputs, Level 1 [Member] | Reported Value Measurement [Member] | Brokered Deposits [Member]
Deposits	229,909,000	183,600,000
Fair Value, Inputs, Level 1 [Member] | Reported Value Measurement [Member] | Cash and Due from Banks [Member]
Cash and cash equivalents	2,693,000	7,381,000
Fair Value, Inputs, Level 1 [Member] | Reported Value Measurement [Member] | Interest-bearing Deposits [Member]
Cash and cash equivalents	36,711,000	59,056,000
Fair Value, Inputs, Level 1 [Member] | Estimate of Fair Value Measurement [Member] | Brokered Deposits [Member]
Deposits	230,073,000	183,120,000
Fair Value, Inputs, Level 1 [Member] | Estimate of Fair Value Measurement [Member] | Cash and Due from Banks [Member]
Cash and cash equivalents	2,693,000	7,381,000
Fair Value, Inputs, Level 1 [Member] | Estimate of Fair Value Measurement [Member] | Interest-bearing Deposits [Member]
Cash and cash equivalents	36,711,000	59,056,000
Fair Value, Inputs, Level 2 [Member] | Reported Value Measurement [Member]
Fair value, total	48,317,000	39,496,000
Other investments	4,450,000	4,963,000
Federal Reserve Bank stock	2,897,000	2,866,000
Federal Home Loan Bank stock	4,477,000	4,928,000
SBA loans held for sale	15,282,000
Accrued interest receivable	3,603,000	3,766,000
FHLB borrowings	100,000,000	100,000,000
Senior notes	11,853,000	11,778,000
Subordinated debt	9,752,000	9,723,000
Junior subordinated debt owed to unconsolidated trust	8,102,000	8,094,000
Accrued interest payable	1,971,000	1,605,000
Interest rate swap liability	694,000	286,000
Fair Value, Inputs, Level 2 [Member] | Reported Value Measurement [Member] | Demand Deposits [Member]
Deposits	88,135,000	84,471,000
Fair Value, Inputs, Level 2 [Member] | Reported Value Measurement [Member] | Savings Deposits [Member]
Deposits	64,020,000	81,912,000
Fair Value, Inputs, Level 2 [Member] | Reported Value Measurement [Member] | Money Market Deposits [Member]
Deposits	99,115,000	85,197,000
Fair Value, Inputs, Level 2 [Member] | Reported Value Measurement [Member] | Negotiable Order of Withdrawal (NOW) Accounts [Member]
Deposits	26,864,000	26,100,000
Fair Value, Inputs, Level 2 [Member] | Reported Value Measurement [Member] | Time Deposits [Member]
Deposits	261,492,000	282,001,000
Fair Value, Inputs, Level 2 [Member] | Estimate of Fair Value Measurement [Member]
Fair value, total	48,317,000	39,496,000
Other investments	4,450,000	4,963,000
Federal Reserve Bank stock	2,897,000	2,866,000
Federal Home Loan Bank stock	4,477,000	4,928,000
SBA loans held for sale	16,733,000
Accrued interest receivable	3,603,000	3,766,000
FHLB borrowings	103,962,000	101,369,000
Senior notes	11,722,000	11,293,000
Subordinated debt	9,747,000	9,348,000
Junior subordinated debt owed to unconsolidated trust	8,102,000	8,094,000
Accrued interest payable	1,971,000	1,605,000
Interest rate swap liability	694,000	286,000
Fair Value, Inputs, Level 2 [Member] | Estimate of Fair Value Measurement [Member] | Demand Deposits [Member]
Deposits	88,135,000	84,471,000
Fair Value, Inputs, Level 2 [Member] | Estimate of Fair Value Measurement [Member] | Savings Deposits [Member]
Deposits	64,020,000	81,912,000
Fair Value, Inputs, Level 2 [Member] | Estimate of Fair Value Measurement [Member] | Money Market Deposits [Member]
Deposits	99,115,000	85,197,000
Fair Value, Inputs, Level 2 [Member] | Estimate of Fair Value Measurement [Member] | Negotiable Order of Withdrawal (NOW) Accounts [Member]
Deposits	26,864,000	26,100,000
Fair Value, Inputs, Level 2 [Member] | Estimate of Fair Value Measurement [Member] | Time Deposits [Member]
Deposits	261,914,000	280,538,000
Fair Value, Inputs, Level 3 [Member]
SBA servicing assets		37,000
Fair Value, Inputs, Level 3 [Member] | Reported Value Measurement [Member]
Loans receivable, net	802,049,000	772,767,000
SBA servicing assets	201,000	37,000
Note payable	1,193,000	1,388,000
Contingent consideration liability	86,000	707,000
Fair Value, Inputs, Level 3 [Member] | Estimate of Fair Value Measurement [Member]
Loans receivable, net	793,559,000	762,581,000
SBA servicing assets	280,000	37,000
Note payable	1,129,000	1,239,000
Contingent consideration liability	$ 86,000	$ 707,000