This package has changed maintainer in Fedora. Reassigning to the new maintainer of this component.

Hi, I’m the new volunteer maintainer for Fedora and EPEL. This package was orphaned in Fedora, and I brought it back from retirement and updated it to the latest version there. I also plan to package the latest version for EPEL8.

For EPEL7, I can update the package to a later version, but I must ensure it is API-compatible; the https://fedoraproject.org/wiki/EPEL_incompatible_upgrades_policy only covers security updates that cannot be handled by backporting.

I’m reviewing the source diff from 0.4.2 to 0.6.10 to check for API breakage between minor versions. If there is some, I might need your help to find which versions do or do not fix the problem.

> This package was orphaned in Fedora, and I brought it back from retirement and updated it to the latest version there.

I meant to say that it was retired along with all Python 2 only packages, and I brought it back as a Python 3 only package. Anyway…

It looks like the updates from 0.4.x to 0.5.x and 0.6.x have been intended to be API-compatibile. However, 0.4.7 and later require lxml 3.3.0 or later, and EL7 has only python-lxml-3.2.1. It seems like this requirement bump was intended to fix some issues, although I cannot find a detailed explanation:

  * https://github.com/ncclient/ncclient/issues/96
  * https://github.com/ncclient/ncclient/issues/91

If I loosened the requirement, I am not confident that it would not cause some errors at runtime; while the difference from 0.4.6 to 0.4.7 was mostly the requirement bump, perhaps later versions do use newer APIs.

-----

I’m hoping you can help me figure out if the fix was in lxml or ncclient.

Do you still have the problem when you do “pip install ncclient==0.4.2” (preferably in a virtualenv), and it installs a later lxml (lxml-4.6.2 when I tried)?

If so, do you also have it for  “pip install ncclient==0.4.6”?

Is your application Python 2 only? A separate python3-ncclient for EPEL7 with a later version is still a possibility even if the Python 2 package cannot be updated.

I've made an attempt to test what you requested, but I'm not having any real success.

The code snippet above still behaves the same way when installing python-ncclient with yum, which I guess is expected as the package is still the same.

I made an attempt to install ncclient in python venv's per your suggestion, but I was unable to do any successful pip install.

Using python2 on centos7.9 I don't have access to pip at all:

    [root@b4b9c334dd20 py2]# source ncclient042/bin/activate                                                                                                                                                                                    
    (ncclient042) [root@b4b9c334dd20 py2]# pip install ncclient==0.4.2                                                                                                                                                                          
    bash: pip: command not found
    (ncclient042) [root@b4b9c334dd20 py2]# deactivate

Going to python3 (still on centos7.9) I have access to pip, but am unable to install ncclient:

    (ncclient042) [root@b4b9c334dd20 py3]# pip install ncclient==0.4.2
    Collecting ncclient==0.4.2
      Downloading ncclient-0.4.2.tar.gz (32 kB)
        ERROR: Command errored out with exit status 1:
         command: /root/py3/ncclient042/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ltkmgmqu/ncclient_ee89d0996d2e493fb86b244b0477dc87/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ltkmgmqu/ncclient_ee89d0996d2e493fb86b244b0477dc87/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-5dhi1nqx
             cwd: /tmp/pip-install-ltkmgmqu/ncclient_ee89d0996d2e493fb86b244b0477dc87/
        Complete output (6 lines):
        Traceback (most recent call last):
          File "<string>", line 1, in <module>
          File "/tmp/pip-install-ltkmgmqu/ncclient_ee89d0996d2e493fb86b244b0477dc87/setup.py", line 24
            print "Sorry, Python 3 is not supported (yet)"
                                                         ^
        SyntaxError: invalid syntax
        ----------------------------------------
    WARNING: Discarding https://files.pythonhosted.org/packages/e1/b1/2f1e912cd7d1aafcae0490339283991c1abd8a2b8c3b244bd35ec9bfefb0/ncclient-0.4.2.tar.gz#sha256=336ce8e8dbf28cb84ff6d0170b3430d9a7648014cf789bf45b5c7c2b6fe00dea (from https://pypi.org/simple/ncclient/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
    ERROR: Could not find a version that satisfies the requirement ncclient==0.4.2
    ERROR: No matching distribution found for ncclient==0.4.2
    (ncclient042) [root@b4b9c334dd20 py3]# deactivate 

0.4.6 gives the same error (but with different hashes of course).

Have I misinterpreted what you needed from me? Should I do something differently?

That’s fine, I think. You’ve proven that version 0.4.6 is not adequate, and the python36-lxml in EPEL7 is newer (4.5.3) so it is not that.

-----

I’m sorry to have inadvertently prompted you to “pip install” as root. Here’s what a safer test could look like (using Python 2 on a real CentOS 7 box):

> $ sudo dnf install python2-pip python-virtualenv
> [lots of output]
> Installed:
>   python2-pip-8.1.2-14.el7.noarch
>   python-virtualenv-15.1.0-4.el7_7.noarch
>   python-devel-2.7.5-90.el7.x86_64
> 
> $ python2 -m virtualenv _e
> New python executable in /home/ben/_e/bin/python2
> Also creating executable in /home/ben/_e/bin/python
> Installing setuptools, pip, wheel...done.
> 
> $ . _e/bin/activate
> 
> (_e) $ pip install ncclient==0.4.6
> [lots of output]
> Successfully installed bcrypt-3.1.7 cffi-1.14.5 cryptography-3.3.2 enum34-1.1.10 ipaddress-1.0.23 lxml-4.6.2 ncclient-0.4.6 paramiko-2.7.2 pycparser-2.20 pynacl-1.4.0 six-1.15.0
> You are using pip version 9.0.1, however version 21.0.1 is available.
> You should consider upgrading via the 'pip install --upgrade pip' command.
> 
> (_e) $ [whatever your test is]
> 
> (_e) $ deactivate
> 
> $ rm -rf _e

-----

Given that 0.4.6 does not fix it, I am not sure I can fix this by updating the version in EPEL7 for Python 2 without too much risk of breaking other people’s working code.

I can look in the source code and try to identify a specific fix that seems like it would fix this traceback, and backport it if I find one.

If you have a need for a Python 3 version, I can probably get a separate EPEL7-only package created with the latest version. I’ll only go through with that if is actually helpful, though.

Since the exception is repackaged from with only the message preserved from the original, there is no information about exactly where it is coming from. I’m having a hard time guessing what’s happening just from skimming the code.

There are a couple of things that might help, if you have the time:

1. Add

> import logging
> logging.basicConfig(level=logging.DEBUG)

to your test, and try it again. This should give some idea of what is happening before the exception, as in https://github.com/ncclient/ncclient/issues/175.

2. Bisect the available versions to find the oldest one that works, using pip and virtualenv as shown in my previous comment; this would be a big help in narrowing down what changed.

> I’m sorry to have inadvertently prompted you to “pip install” as root.

No worries, it was a burner container on a burner server on a test network.

> If you have a need for a Python 3 version, I can probably get a separate EPEL7-only package created with the latest version.

The code using the module is python3 now, so that package would let me deprecate a venv. That would be beneficial, but not critical. Feel bad putting that maintenance burden on you, but if you’re up for it I’d use it. 

> 2. Bisect the available versions to find the oldest one that works

Might be a while before I’m able to do that.

Looks like this is a WONTFIX for Python 2, unless someone has time to figure out a patch that can be backported. I may or may not be able to do that if you are able to capture debug logging output.

Meanwhile, I am building a separate Python 3 package for EPEL7 that uses the latest release of ncclient. (Again, the versions of ncclient that work with the old version of lxml available for Python 2 in EL7, and the versions that support Python 3, are mutually exclusive, so an update to the base package was not an option.) You should soon see a comment in this bug linking to that package for testing.

FEDORA-EPEL-2021-6cc540e7b2 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-6cc540e7b2

FEDORA-EPEL-2021-6cc540e7b2 has been pushed to the Fedora EPEL 7 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2021-6cc540e7b2

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

I managed to do some more testing earlier than I anticipated, but the results are nothing much to write home about.

I added logging, to the top of the test script, as you suggested:

> #!/usr/bin/python
> from ncclient import manager
> import logging
> logging.basicConfig(level=logging.DEBUG)
> with manager.connect(host='[...]

But the output I got from the script was the same as before.

I thought maybe the log ended up somewhere I couldn't find it so I tried sending it to a file instead like so:

> import logging
> logging.basicConfig(filename='debug.log', level=logging.DEBUG)

A file is created, but no data is sent to it.

Maybe the currently packaged version of ncclient is so old that it is not producing any log output?

> But the output I got from the script was the same as before.

> Maybe the currently packaged version of ncclient is so old that it is not producing any log output?

Thanks for looking at it. That’s interesting. Your theory seems reasonable.

I don’t think I’m going to spend any more time trying to fix Python 2 package unless some new information shows up in this issue.

FEDORA-EPEL-2021-6cc540e7b2 has been pushed to the Fedora EPEL 7 stable repository.
If problem still persists, please make note of it in this bug report.

I just circled back to this and attempted installing the ncclient package for python3 from epel. Seems there are still room for some improvements.

> # yum install python36-ncclient.noarch
> [...]
> Downloading packages:
> python36-ncclient-0.6.12-2.el7.noarch.rpm                                          | 135 kB  00:00:00
> Running transaction check
> Running transaction test
> Transaction test succeeded
> Running transaction
>   Installing : python36-ncclient-0.6.12-2.el7.noarch
>   Verifying  : python36-ncclient-0.6.12-2.el7.noarch
> 
> Installed:
>   python36-ncclient.noarch 0:0.6.12-2.el7
> 
> Complete!
> [matfra@nanna ~]$ python3
> Python 3.6.8 (default, Nov 16 2020, 16:55:22) 
> [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> >>> from ncclient import manager
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/usr/lib/python3.6/site-packages/ncclient/manager.py", line 21, in <module>
>     from ncclient import operations
>   File "/usr/lib/python3.6/site-packages/ncclient/operations/__init__.py", line 16, in <module>
>     from ncclient.operations.rpc import RPC, RPCReply, RPCError, RaiseMode, GenericRPC
>   File "/usr/lib/python3.6/site-packages/ncclient/operations/rpc.py", line 18, in <module>
>     from ncclient.xml_ import *
>   File "/usr/lib/python3.6/site-packages/ncclient/xml_.py", line 25, in <module>
>     from lxml import etree
> ModuleNotFoundError: No module named 'lxml'
 
> # yum install python36-lxml
> [...]
> Running transaction
>   Installing : python36-lxml-4.2.5-4.el7.x86_64
>   Verifying  : python36-lxml-4.2.5-4.el7.x86_64
> 
> Installed:
>   python36-lxml.x86_64 0:4.2.5-4.el7
> 
> Complete!
> [matfra@nanna ~]$ python3
> Python 3.6.8 (default, Nov 16 2020, 16:55:22) 
> [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)] on linux
> Type "help", "copyright", "credits" or "license" for more information.
> >>> from ncclient import manager
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/usr/lib/python3.6/site-packages/ncclient/manager.py", line 21, in <module>
>     from ncclient import operations
>   File "/usr/lib/python3.6/site-packages/ncclient/operations/__init__.py", line 16, in <module>
>     from ncclient.operations.rpc import RPC, RPCReply, RPCError, RaiseMode, GenericRPC
>   File "/usr/lib/python3.6/site-packages/ncclient/operations/rpc.py", line 20, in <module>
>     from ncclient.transport import SessionListener
>   File "/usr/lib/python3.6/site-packages/ncclient/transport/__init__.py", line 18, in <module>
>     from ncclient.transport.ssh import SSHSession
>   File "/usr/lib/python3.6/site-packages/ncclient/transport/ssh.py", line 34, in <module>
>     import paramiko
> ModuleNotFoundError: No module named 'paramiko'

After installing python36-paramiko the module imports, so I guess both the lxml-package and the parameko-package should be in the ncclient-package dependency list.

Thanks for the feedback. I’m away from a suitable computer, but I’ll fix this with an update in the next week or two.

It’s easy to forget dependencies when backporting to EPEL7 since all other EPEL’s and Fedoras generate them automatically. Ideally I should always do a test-install and import for a new backport, but I obviously didn’t.

FEDORA-EPEL-2022-b63d2bd67f has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-b63d2bd67f

FEDORA-EPEL-2022-b63d2bd67f has been pushed to the Fedora EPEL 7 testing repository.

You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2022-b63d2bd67f

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-EPEL-2022-b63d2bd67f has been pushed to the Fedora EPEL 7 stable repository.
If problem still persists, please make note of it in this bug report.