Note 13 - Fair Value and Interest Rate Risk - Financial Assets and Liabilities (Details) - USD ($)	Mar. 31, 2020	Dec. 31, 2019
Fair value, total	$ 44,830,000	$ 48,317,000
SBA servicing assets	198,000
Interest rate swap receivable	1,295,000	694,000
Reported Value Measurement [Member]
Financial assets, total	940,025,000	921,374,000
Financial liabilities, total	927,035,000	903,186,000
Estimate of Fair Value Measurement [Member]
Financial assets, total	914,338,000	914,414,000
Financial liabilities, total	938,481,000	907,534,000
Fair Value, Inputs, Level 1 [Member] | Reported Value Measurement [Member] | Brokered Deposits [Member]
Deposits	225,415,000	229,909,000
Fair Value, Inputs, Level 1 [Member] | Reported Value Measurement [Member] | Cash and Due from Banks [Member]
Cash and cash equivalents	1,806,000	2,693,000
Fair Value, Inputs, Level 1 [Member] | Reported Value Measurement [Member] | Interest-bearing Deposits [Member]
Cash and cash equivalents	50,350,000	36,711,000
Fair Value, Inputs, Level 1 [Member] | Estimate of Fair Value Measurement [Member] | Brokered Deposits [Member]
Deposits	226,786,000	230,073,000
Fair Value, Inputs, Level 1 [Member] | Estimate of Fair Value Measurement [Member] | Cash and Due from Banks [Member]
Cash and cash equivalents	1,806,000	2,693,000
Fair Value, Inputs, Level 1 [Member] | Estimate of Fair Value Measurement [Member] | Interest-bearing Deposits [Member]
Cash and cash equivalents	50,350,000	36,711,000
Fair Value, Inputs, Level 2 [Member] | Reported Value Measurement [Member]
Fair value, total	44,830,000	48,317,000
Other investments	4,450,000	4,450,000
Federal Reserve Bank stock	2,897,000	2,897,000
Federal Home Loan Bank stock	4,477,000	4,477,000
SBA loans held for sale	17,996,000	15,282,000
Accrued interest receivable	3,801,000	3,603,000
FHLB borrowings	90,000,000	100,000,000
Senior notes	11,871,000	11,853,000
Subordinated debt	9,760,000	9,752,000
Junior subordinated debt owed to unconsolidated trust	8,104,000	8,102,000
Accrued interest payable	1,562,000	1,971,000
Interest rate swap liability	1,295,000	694,000
Fair Value, Inputs, Level 2 [Member] | Reported Value Measurement [Member] | Demand Deposits [Member]
Deposits	83,583,000	88,135,000
Fair Value, Inputs, Level 2 [Member] | Reported Value Measurement [Member] | Savings Deposits [Member]
Deposits	59,567,000	64,020,000
Fair Value, Inputs, Level 2 [Member] | Reported Value Measurement [Member] | Money Market Deposits [Member]
Deposits	132,629,000	99,115,000
Fair Value, Inputs, Level 2 [Member] | Reported Value Measurement [Member] | Negotiable Order of Withdrawal (NOW) Accounts [Member]
Deposits	28,265,000	26,864,000
Fair Value, Inputs, Level 2 [Member] | Reported Value Measurement [Member] | Time Deposits [Member]
Deposits	273,755,000	261,492,000
Fair Value, Inputs, Level 2 [Member] | Estimate of Fair Value Measurement [Member]
Fair value, total	44,830,000	48,317,000
Other investments	4,450,000	4,450,000
Federal Reserve Bank stock	2,897,000	2,897,000
Federal Home Loan Bank stock	4,477,000	4,477,000
SBA loans held for sale	19,672,000	16,733,000
Accrued interest receivable	3,801,000	3,603,000
FHLB borrowings	97,410,000	103,962,000
Senior notes	11,985,000	11,722,000
Subordinated debt	10,117,000	9,747,000
Junior subordinated debt owed to unconsolidated trust	8,104,000	8,102,000
Accrued interest payable	1,562,000	1,971,000
Interest rate swap liability	1,295,000	694,000
Fair Value, Inputs, Level 2 [Member] | Estimate of Fair Value Measurement [Member] | Demand Deposits [Member]
Deposits	83,583,000	88,135,000
Fair Value, Inputs, Level 2 [Member] | Estimate of Fair Value Measurement [Member] | Savings Deposits [Member]
Deposits	59,567,000	64,020,000
Fair Value, Inputs, Level 2 [Member] | Estimate of Fair Value Measurement [Member] | Money Market Deposits [Member]
Deposits	132,629,000	99,115,000
Fair Value, Inputs, Level 2 [Member] | Estimate of Fair Value Measurement [Member] | Negotiable Order of Withdrawal (NOW) Accounts [Member]
Deposits	28,265,000	26,864,000
Fair Value, Inputs, Level 2 [Member] | Estimate of Fair Value Measurement [Member] | Time Deposits [Member]
Deposits	275,934,000	261,914,000
Fair Value, Inputs, Level 3 [Member]
SBA servicing assets		280,000
Fair Value, Inputs, Level 3 [Member] | Reported Value Measurement [Member]
Loans receivable, net	807,925,000	802,049,000
SBA servicing assets	198,000	201,000
Note payable	1,143,000	1,193,000
Contingent consideration liability	86,000	86,000
Fair Value, Inputs, Level 3 [Member] | Estimate of Fair Value Measurement [Member]
Loans receivable, net	780,488,000	793,559,000
SBA servicing assets	272,000	280,000
Note payable	1,158,000	1,129,000
Contingent consideration liability	$ 86,000	$ 86,000