Bug 2143974
| Summary: | ansible-core vendors resolvelib but still references to it in its requirements.txt, breaking pkg_resources | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Evgeni Golov <egolov> |
| Component: | ansible-core | Assignee: | Dimitri Savineau <dsavinea> |
| Status: | CLOSED ERRATA | QA Contact: | Matt Clay <mclay> |
| Severity: | low | Docs Contact: | |
| Priority: | low | ||
| Version: | 8.7 | Keywords: | Triaged |
| Target Milestone: | rc | Flags: | pm-rhel:
mirror+
|
| Target Release: | 8.8 | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | ansible-core-2.14.2-3.el8 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-05-16 08:24:54 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Looking at # cat /usr/lib/python3.9/site-packages/ansible_core-2.13.3-py3.9.egg-info/requires.txt jinja2>=3.0.0 PyYAML>=5.1 cryptography packaging resolvelib<0.9.0,>=0.5.3 and # ls /usr/lib/python3.9/site-packages/ansible/_vendor/ __init__.py Jinja2-3.1.2-py3.9.egg-info MarkupSafe-2.1.0-py3.9.egg-info packaging-20.4-py3.9.egg-info pyparsing-2.4.7-py3.9.egg-info resolvelib jinja2 markupsafe packaging __pycache__ pyparsing.py resolvelib-0.5.4-py3.9.egg-info I guess this also applies to the other requirements? AFAIK requirements in the requires.txt are generated automatically from python package requirements so that file doesn't need to be updated if some dependencies are bundled or not. I'm more curious about the pkg_resources usage in the callback. From what I know, pkg_resources is discouraged in flavor or builtin importlib.metadata and importlib.resources In this situation you could switch from - from pkg_resources import iter_entry_points + from importlib.metadata import entry_points and adapt the rest of the code. With a quick test, I don't have the vendored dependencies issue with importlib.metadata. [1] https://docs.python.org/3/library/importlib.metadata.html#entry-points (In reply to Dimitri Savineau from comment #2) > AFAIK requirements in the requires.txt are generated automatically from > python package requirements so that file doesn't need to be updated if some > dependencies are bundled or not. Yes, this file is autogenerated. Yet it (IMHO) should still reflect the (external) dependencies of the installed software. Given that resolvelib (and others) are vendored in the Ansible RPM (but not if you install Ansible from e.g. PyPI), the contents of that file should also reflect that you removed the external dependency on resolvelib. > I'm more curious about the pkg_resources usage in the callback. > > From what I know, pkg_resources is discouraged in flavor or builtin > importlib.metadata and importlib.resources > > In this situation you could switch from > > - from pkg_resources import iter_entry_points > + from importlib.metadata import entry_points > > and adapt the rest of the code. That does sound correct to me, but it's sadly never as easy as that. The above example was just that, an example. The real issue I am facing is with the community.general.opentelemetry callback, which in turn depends on the opentelementry library and *that* uses pkg_resources. The full traceback then reads like this: Traceback (most recent call last): File "/usr/lib/python3.9/site-packages/ansible/cli/__init__.py", line 601, in cli_executor exit_code = cli.run() File "/usr/lib/python3.9/site-packages/ansible/cli/playbook.py", line 143, in run results = pbex.run() File "/usr/lib/python3.9/site-packages/ansible/executor/playbook_executor.py", line 119, in run self._tqm.load_callbacks() File "/usr/lib/python3.9/site-packages/ansible/executor/task_queue_manager.py", line 185, in load_callbacks plugin = callback_loader.get(c, class_only=True) File "/usr/lib/python3.9/site-packages/ansible/plugins/loader.py", line 825, in get return self.get_with_context(name, *args, **kwargs).object File "/usr/lib/python3.9/site-packages/ansible/plugins/loader.py", line 845, in get_with_context self._module_cache[path] = self._load_module_source(name, path) File "/usr/lib/python3.9/site-packages/ansible/plugins/loader.py", line 810, in _load_module_source spec.loader.exec_module(module) File "<frozen importlib._bootstrap_external>", line 850, in exec_module File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed File "/usr/lib/python3.9/site-packages/ansible_collections/community/general/plugins/callback/opentelemetry.py", line 103, in <module> from opentelemetry import trace File "/home/user/.local/lib/python3.9/site-packages/opentelemetry/trace/__init__.py", line 87, in <module> from opentelemetry import context as context_api File "/home/user/.local/lib/python3.9/site-packages/opentelemetry/context/__init__.py", line 22, in <module> from pkg_resources import iter_entry_points File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3239, in <module> def _initialize_master_working_set(): File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3222, in _call_aside f(*args, **kwargs) File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 3251, in _initialize_master_working_set working_set = WorkingSet._build_master() File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 567, in _build_master ws.require(__requires__) File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 884, in require needed = self.resolve(parse_requirements(requirements)) File "/usr/lib/python3.9/site-packages/pkg_resources/__init__.py", line 770, in resolve raise DistributionNotFound(req, requirers) pkg_resources.DistributionNotFound: The 'resolvelib<0.9.0,>=0.5.3' distribution was not found and is required by ansible-core And I bet there are many more libraries still using pkg_resources, which, while deprecated, ought to still work. > The real issue I am facing is with the community.general.opentelemetry callback, which in turn depends on the opentelementry library and *that* uses pkg_resources. The main issue I have with this is that the ansible-core support scope in RHEL is limited to the rhel-system-roles RPM package (which doesn't use pkg_resources) so the community.general collection isn't supported. That said, I'll discuss with the upstream core team to see if this is a RPM related issue, an issue with the way the callbacks are loaded or other. FYI, ansible loads the path to vendored dependencies [1][2] so from the custom callback you have access to the vendored dependencies. The issue is likely with the pkg_resources usage as it's not using the ansible vendoring path. [1] https://github.com/ansible/ansible/blob/stable-2.13/lib/ansible/__init__.py#L22-L23 [2] https://github.com/ansible/ansible/blob/stable-2.13/lib/ansible/_vendor/__init__.py Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (ansible-core bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2023:2762 |
Description of problem: When trying to use "pkg_resources" *inside* Ansible (e.g. in a callback plugin), the plugin can't be loaded as pkg_resources detects that ansible-core has a dependency on resolvelib, but that is not installed (because it is vendored). Version-Release number of selected component (if applicable): ansible-core-2.13.3-1.el8.x86_64 (RHEL 8.7) ansible-core-2.13.5-1.el8.x86_64 (CentOS Stream 8) How reproducible: 100% Steps to Reproduce: 1. create trivial playbook: vim test.yml --- - hosts: localhost tasks: - name: test debug: msg: test 2. create trivial callback: mkdir callback_plugins && vim callback_plugins/demo.py from pkg_resources import iter_entry_points 3. ANSIBLE_CALLBACKS_ENABLED='demo' ansible-playbook test.yml Actual results: # ANSIBLE_CALLBACKS_ENABLED='demo' ansible-playbook t.yml [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Skipping plugin (/root/ansible-wtf/callback_plugins/demo.py) as it seems to be invalid: The 'resolvelib<0.9.0,>=0.5.3' distribution was not found and is required by ansible-core ERROR! Unexpected Exception, this is probably a bug: The 'resolvelib<0.9.0,>=0.5.3' distribution was not found and is required by ansible-core Expected results: No error :) Additional info: # grep resolv /usr/lib/python3.9/site-packages/ansible_core-2.13.3-py3.9.egg-info/requires.txt resolvelib<0.9.0,>=0.5.3