Note 8 - Fair Value of Financial Instruments - Financial Assets Measured on a Recurring Basis (Details) - USD ($) $ in Thousands	Sep. 30, 2020	Jun. 30, 2020	Dec. 31, 2019	Sep. 30, 2019	Jun. 30, 2019	Dec. 31, 2018
Total, available for sale, fair value	$ 440,655		$ 539,042
SBA Servicing Assets	4,634	$ 4,604	4,447	$ 4,504	$ 4,593	$ 4,785
US Government Agencies Debt Securities [Member]
Total, available for sale, fair value	33,372		38,305
Collateralized Mortgage Obligations [Member]
Total, available for sale, fair value	203,802		331,438
Agency Mortgage-backed Securities [Member]
Total, available for sale, fair value	131,203		98,937
US States and Political Subdivisions Debt Securities [Member]
Total, available for sale, fair value	2,662		4,082
Corporate Debt Securities [Member]
Total, available for sale, fair value	69,616		66,280
Fair Value, Recurring [Member]
Total, available for sale, fair value	440,655		539,042
Mortgages Held-for-sale, Fair Value Disclosure	41,683		10,345
SBA Servicing Assets	4,634		4,447
Fair Value, Recurring [Member] | Fair Value, Inputs, Level 1 [Member]
Total, available for sale, fair value	0		0
Mortgages Held-for-sale, Fair Value Disclosure	0		0
SBA Servicing Assets	0		0
Fair Value, Recurring [Member] | Fair Value, Inputs, Level 2 [Member]
Total, available for sale, fair value	438,024		536,222
Mortgages Held-for-sale, Fair Value Disclosure	41,683		10,345
SBA Servicing Assets	0		0
Fair Value, Recurring [Member] | Fair Value, Inputs, Level 3 [Member]
Total, available for sale, fair value	2,631		2,820
Mortgages Held-for-sale, Fair Value Disclosure	0		0
SBA Servicing Assets	4,634		4,447
Fair Value, Recurring [Member] | US Government Agencies Debt Securities [Member]
Total, available for sale, fair value	33,372		38,305
Fair Value, Recurring [Member] | US Government Agencies Debt Securities [Member] | Fair Value, Inputs, Level 1 [Member]
Total, available for sale, fair value	0		0
Fair Value, Recurring [Member] | US Government Agencies Debt Securities [Member] | Fair Value, Inputs, Level 2 [Member]
Total, available for sale, fair value	33,372		38,305
Fair Value, Recurring [Member] | US Government Agencies Debt Securities [Member] | Fair Value, Inputs, Level 3 [Member]
Total, available for sale, fair value	0		0
Fair Value, Recurring [Member] | Collateralized Mortgage Obligations [Member]
Total, available for sale, fair value	203,802		331,438
Fair Value, Recurring [Member] | Collateralized Mortgage Obligations [Member] | Fair Value, Inputs, Level 1 [Member]
Total, available for sale, fair value	0		0
Fair Value, Recurring [Member] | Collateralized Mortgage Obligations [Member] | Fair Value, Inputs, Level 2 [Member]
Total, available for sale, fair value	203,802		331,438
Fair Value, Recurring [Member] | Collateralized Mortgage Obligations [Member] | Fair Value, Inputs, Level 3 [Member]
Total, available for sale, fair value	0		0
Fair Value, Recurring [Member] | Agency Mortgage-backed Securities [Member]
Total, available for sale, fair value	131,203		98,937
Fair Value, Recurring [Member] | Agency Mortgage-backed Securities [Member] | Fair Value, Inputs, Level 1 [Member]
Total, available for sale, fair value	0		0
Fair Value, Recurring [Member] | Agency Mortgage-backed Securities [Member] | Fair Value, Inputs, Level 2 [Member]
Total, available for sale, fair value	131,203		98,937
Fair Value, Recurring [Member] | Agency Mortgage-backed Securities [Member] | Fair Value, Inputs, Level 3 [Member]
Total, available for sale, fair value	0		0
Fair Value, Recurring [Member] | US States and Political Subdivisions Debt Securities [Member]
Total, available for sale, fair value	2,662		4,082
Fair Value, Recurring [Member] | US States and Political Subdivisions Debt Securities [Member] | Fair Value, Inputs, Level 1 [Member]
Total, available for sale, fair value	0		0
Fair Value, Recurring [Member] | US States and Political Subdivisions Debt Securities [Member] | Fair Value, Inputs, Level 2 [Member]
Total, available for sale, fair value	2,662		4,082
Fair Value, Recurring [Member] | US States and Political Subdivisions Debt Securities [Member] | Fair Value, Inputs, Level 3 [Member]
Total, available for sale, fair value	0		0
Fair Value, Recurring [Member] | Corporate Debt Securities [Member]
Total, available for sale, fair value	69,616		66,280
Fair Value, Recurring [Member] | Corporate Debt Securities [Member] | Fair Value, Inputs, Level 1 [Member]
Total, available for sale, fair value	0		0
Fair Value, Recurring [Member] | Corporate Debt Securities [Member] | Fair Value, Inputs, Level 2 [Member]
Total, available for sale, fair value	66,985		63,460
Fair Value, Recurring [Member] | Corporate Debt Securities [Member] | Fair Value, Inputs, Level 3 [Member]
Total, available for sale, fair value	2,631		2,820
Fair Value, Recurring [Member] | Interest Rate Lock Commitments [Member]
Derivative Asset	1,451		362
Derivative Liability	1		0
Fair Value, Recurring [Member] | Interest Rate Lock Commitments [Member] | Fair Value, Inputs, Level 1 [Member]
Derivative Asset	0		0
Derivative Liability	0		0
Fair Value, Recurring [Member] | Interest Rate Lock Commitments [Member] | Fair Value, Inputs, Level 2 [Member]
Derivative Asset	1,451		362
Derivative Liability	1		0
Fair Value, Recurring [Member] | Interest Rate Lock Commitments [Member] | Fair Value, Inputs, Level 3 [Member]
Derivative Asset	0		0
Derivative Liability	0		0
Fair Value, Recurring [Member] | Best Efforts Forward Loan sales Commitments [Member]
Derivative Asset	0		4
Derivative Liability	554		133
Fair Value, Recurring [Member] | Best Efforts Forward Loan sales Commitments [Member] | Fair Value, Inputs, Level 1 [Member]
Derivative Asset	0		0
Derivative Liability	0		0
Fair Value, Recurring [Member] | Best Efforts Forward Loan sales Commitments [Member] | Fair Value, Inputs, Level 2 [Member]
Derivative Asset	0		4
Derivative Liability	554		133
Fair Value, Recurring [Member] | Best Efforts Forward Loan sales Commitments [Member] | Fair Value, Inputs, Level 3 [Member]
Derivative Asset	0		0
Derivative Liability	0		0
Fair Value, Recurring [Member] | Mandatory Forward Loan Sales Commitments [Member]
Derivative Asset	0		2
Derivative Liability	525		83
Fair Value, Recurring [Member] | Mandatory Forward Loan Sales Commitments [Member] | Fair Value, Inputs, Level 1 [Member]
Derivative Asset	0		0
Derivative Liability	0		0
Fair Value, Recurring [Member] | Mandatory Forward Loan Sales Commitments [Member] | Fair Value, Inputs, Level 2 [Member]
Derivative Asset	0		2
Derivative Liability	525		83
Fair Value, Recurring [Member] | Mandatory Forward Loan Sales Commitments [Member] | Fair Value, Inputs, Level 3 [Member]
Derivative Asset	0		0
Derivative Liability	$ 0		$ 0