Note 15 - Fair Value Measurements and Fair Values of Financial Instruments - Financial Assets Measured at on a Recurring Basis (Details) - USD ($) $ in Thousands	Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017	Dec. 31, 2016
Total, available for sale, fair value	$ 539,042	$ 321,014
SBA Servicing Assets	4,447	4,785	$ 5,243	$ 5,352
US Government Agencies Debt Securities [Member]
Total, available for sale, fair value	38,305
Collateralized Mortgage Obligations [Member]
Total, available for sale, fair value	331,438	196,259
Agency Mortgage-backed Securities [Member]
Total, available for sale, fair value	98,937	38,499
US States and Political Subdivisions Debt Securities [Member]
Total, available for sale, fair value	4,082	20,639
Corporate Debt Securities [Member]
Total, available for sale, fair value	66,280	59,274
Asset-backed Securities [Member]
Total, available for sale, fair value		6,343
Fair Value, Recurring [Member]
Total, available for sale, fair value	539,042	321,014
Mortgages Held-for-sale, Fair Value Disclosure	10,345	20,887
SBA Servicing Assets	4,447	4,785
Fair Value, Recurring [Member] | Fair Value, Inputs, Level 1 [Member]
Total, available for sale, fair value
Mortgages Held-for-sale, Fair Value Disclosure
SBA Servicing Assets
Fair Value, Recurring [Member] | Fair Value, Inputs, Level 2 [Member]
Total, available for sale, fair value	536,222	317,945
Mortgages Held-for-sale, Fair Value Disclosure	10,345	20,887
SBA Servicing Assets
Fair Value, Recurring [Member] | Fair Value, Inputs, Level 3 [Member]
Total, available for sale, fair value	2,820	3,069
Mortgages Held-for-sale, Fair Value Disclosure
SBA Servicing Assets	4,447	4,785
Fair Value, Recurring [Member] | US Government Agencies Debt Securities [Member]
Total, available for sale, fair value	38,305
Fair Value, Recurring [Member] | US Government Agencies Debt Securities [Member] | Fair Value, Inputs, Level 1 [Member]
Total, available for sale, fair value
Fair Value, Recurring [Member] | US Government Agencies Debt Securities [Member] | Fair Value, Inputs, Level 2 [Member]
Total, available for sale, fair value	38,305
Fair Value, Recurring [Member] | US Government Agencies Debt Securities [Member] | Fair Value, Inputs, Level 3 [Member]
Total, available for sale, fair value
Fair Value, Recurring [Member] | Collateralized Mortgage Obligations [Member]
Total, available for sale, fair value	331,438	196,259
Fair Value, Recurring [Member] | Collateralized Mortgage Obligations [Member] | Fair Value, Inputs, Level 1 [Member]
Total, available for sale, fair value
Fair Value, Recurring [Member] | Collateralized Mortgage Obligations [Member] | Fair Value, Inputs, Level 2 [Member]
Total, available for sale, fair value	331,438	196,259
Fair Value, Recurring [Member] | Collateralized Mortgage Obligations [Member] | Fair Value, Inputs, Level 3 [Member]
Total, available for sale, fair value
Fair Value, Recurring [Member] | Agency Mortgage-backed Securities [Member]
Total, available for sale, fair value	98,937	38,499
Fair Value, Recurring [Member] | Agency Mortgage-backed Securities [Member] | Fair Value, Inputs, Level 1 [Member]
Total, available for sale, fair value
Fair Value, Recurring [Member] | Agency Mortgage-backed Securities [Member] | Fair Value, Inputs, Level 2 [Member]
Total, available for sale, fair value	98,937	38,499
Fair Value, Recurring [Member] | Agency Mortgage-backed Securities [Member] | Fair Value, Inputs, Level 3 [Member]
Total, available for sale, fair value
Fair Value, Recurring [Member] | US States and Political Subdivisions Debt Securities [Member]
Total, available for sale, fair value	4,082	20,639
Fair Value, Recurring [Member] | US States and Political Subdivisions Debt Securities [Member] | Fair Value, Inputs, Level 1 [Member]
Total, available for sale, fair value
Fair Value, Recurring [Member] | US States and Political Subdivisions Debt Securities [Member] | Fair Value, Inputs, Level 2 [Member]
Total, available for sale, fair value	4,082	20,639
Fair Value, Recurring [Member] | US States and Political Subdivisions Debt Securities [Member] | Fair Value, Inputs, Level 3 [Member]
Total, available for sale, fair value
Fair Value, Recurring [Member] | Corporate Debt Securities [Member]
Total, available for sale, fair value	66,280	59,274
Fair Value, Recurring [Member] | Corporate Debt Securities [Member] | Fair Value, Inputs, Level 1 [Member]
Total, available for sale, fair value
Fair Value, Recurring [Member] | Corporate Debt Securities [Member] | Fair Value, Inputs, Level 2 [Member]
Total, available for sale, fair value	63,460	56,205
Fair Value, Recurring [Member] | Corporate Debt Securities [Member] | Fair Value, Inputs, Level 3 [Member]
Total, available for sale, fair value	2,820	3,069
Fair Value, Recurring [Member] | Interest Rate Lock Commitments [Member]
Derivative Asset	362	410
Derivative Liability
Fair Value, Recurring [Member] | Interest Rate Lock Commitments [Member] | Fair Value, Inputs, Level 1 [Member]
Derivative Asset
Derivative Liability
Fair Value, Recurring [Member] | Interest Rate Lock Commitments [Member] | Fair Value, Inputs, Level 2 [Member]
Derivative Asset	362	410
Derivative Liability
Fair Value, Recurring [Member] | Interest Rate Lock Commitments [Member] | Fair Value, Inputs, Level 3 [Member]
Derivative Asset
Derivative Liability
Fair Value, Recurring [Member] | Best Efforts Forward Loan sales Commitments [Member]
Derivative Asset	4	5
Derivative Liability	133	138
Fair Value, Recurring [Member] | Best Efforts Forward Loan sales Commitments [Member] | Fair Value, Inputs, Level 1 [Member]
Derivative Asset
Derivative Liability
Fair Value, Recurring [Member] | Best Efforts Forward Loan sales Commitments [Member] | Fair Value, Inputs, Level 2 [Member]
Derivative Asset	4	5
Derivative Liability	133	138
Fair Value, Recurring [Member] | Best Efforts Forward Loan sales Commitments [Member] | Fair Value, Inputs, Level 3 [Member]
Derivative Asset
Derivative Liability
Fair Value, Recurring [Member] | Mandatory Forward Loan Sales Commitments [Member]
Derivative Asset	2	10
Derivative Liability	83	230
Fair Value, Recurring [Member] | Mandatory Forward Loan Sales Commitments [Member] | Fair Value, Inputs, Level 1 [Member]
Derivative Asset
Derivative Liability
Fair Value, Recurring [Member] | Mandatory Forward Loan Sales Commitments [Member] | Fair Value, Inputs, Level 2 [Member]
Derivative Asset	2	10
Derivative Liability	83	230
Fair Value, Recurring [Member] | Mandatory Forward Loan Sales Commitments [Member] | Fair Value, Inputs, Level 3 [Member]
Derivative Asset
Derivative Liability
Fair Value, Recurring [Member] | Asset-backed Securities [Member]
Total, available for sale, fair value		6,343
Fair Value, Recurring [Member] | Asset-backed Securities [Member] | Fair Value, Inputs, Level 1 [Member]
Total, available for sale, fair value
Fair Value, Recurring [Member] | Asset-backed Securities [Member] | Fair Value, Inputs, Level 2 [Member]
Total, available for sale, fair value		6,343
Fair Value, Recurring [Member] | Asset-backed Securities [Member] | Fair Value, Inputs, Level 3 [Member]
Total, available for sale, fair value