Bug 1834529

Summary: python-s3transfer: bad dependency metadata due to bundling futures, jmespath, botocore
Product: Red Hat Enterprise Linux 7 Reporter: Felix Schwarz <fschwarz>
Component: python-s3transferAssignee: Oyvind Albrigtsen <oalbrigt>
Status: CLOSED WONTFIX QA Contact: Brandon Perkins <bperkins>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 7.8CC: bmw, jay.boyd, nb
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-11 07:27:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1834528    

Description Felix Schwarz 2020-05-11 21:00:48 UTC
python-s3transfer 0.1.13-1.el7.0.1 in RHEL 7 contains bad Python-level metadata. It requires Python modules which are bundled.

Why is this a problem?

I can not run the test suite for "python-certbot-dns-route53" in EPEL 7. The certbot plugin depends on python-boto3. When I do "python setup.py test" Python tries to download all missing packages (which fails in koji for obvious reasons). I'm not yet sure if this also a runtime problem.


What should be done?

Remove all bundled packages from the Python-level requirements.


Details:

# cat /usr/lib/python2.7/site-packages/s3transfer-0.1.13-py2.7.egg-info/requires.txt
botocore>=1.3.0,<2.0.0
futures>=2.2.0,<4.0.0

[:python_version=="2.6" or python_version=="2.7"]
futures>=2.2.0,<4.0.0

The package bundles futures, botocore so these entries should be removed as Python-level dependencies (in setup.cfg).

The bundled botocore needs to be fixed (jmespath):

$ cat /usr/lib/fence-agents/bundled/botocore-1.8.35-py2.7.egg-info/requires.txt
jmespath>=0.7.1,<1.0.0
python-dateutil>=1.4,<3.0.0
docutils>=0.10

[:python_version=="2.6"]
ordereddict==1.1
simplejson==3.3.0

Comment 2 Felix Schwarz 2020-05-12 06:58:06 UTC
Btw: also the "futures" requirement looks fishy to me. It should be necessary only for Python 2 but it is listed twice. This is no problem for the present rpm which only supports Python 2 but should be fixed if you want/need to support Python 3 at some point.

Comment 3 Brad Warren 2020-08-12 17:06:13 UTC
This problem breaks the python2-certbot-dns-route53 package.

Certbot uses setuptools to find and load plugins such as python2-certbot-dns-route53 and setuptools verifies that all of the plugin's dependencies are satisified before loading the plugin. As a result, if you install python2-certbot-dns-route53 and run almost any Certbot command such as `certbot plugins`, Certbot will crash with output like:

An unexpected error occurred:
DistributionNotFound: futures>=2.2.0,<4.0.0

Comment 4 Felix Schwarz 2020-08-13 08:30:46 UTC
ugh, that's worse than I expected (though logical in hindsight).

Oyvind, Brandon: Can I help you fixing this somehow?

Comment 5 Oyvind Albrigtsen 2020-08-21 14:11:54 UTC
Where do you get that version from?
$ brew buildinfo python-s3transfer-0.1.13-1.el7.0.1                                                                                                         [11:14:12AM]
No such build: python-s3transfer-0.1.13-1.el7.0.1

This library is only supported for use with the resource-agents and fence-agents-* packages, but if you can make a patch to fix the requires.txt files post-build for the package we can fix it that way.

Comment 6 Felix Schwarz 2020-08-21 14:31:02 UTC
The package is shipped in CentOS 7 in its "base" repo.

# yum info python-s3transfer
Name        : python-s3transfer
Arch        : noarch
Version     : 0.1.13
Release     : 1.el7.0.1
Size        : 2.9 M
Repo        : base/7/x86_64
Summary     : An Amazon S3 Transfer Manager
URL         : https://github.com/boto/s3transfer
License     : ASL 2.0
Description : S3transfer is a Python library for managing Amazon S3 transfers.


> if you can make a patch to fix the requires.txt files post-build for the package we can fix it that way.

Of course I can try to whip up a patch. Just so I don't go in the wrong direction: The easiest way would be to amend "bundled-futures-botocore-jmespath.patch" (https://git.centos.org/rpms/python-s3transfer/blob/c7/f/SOURCES/bundled-futures-botocore-jmespath.patch) so it removes the dependencies directly in setup.py *before* building the actual package. Would that work for you?

Comment 7 Oyvind Albrigtsen 2020-08-21 14:54:31 UTC
I think it'd be better to do it in a separate patch post-build, so the build will still fail properly if we at some point upgrade any of the libs to a version that is incompatible with any of the other libs.

Comment 8 Jay Boyd 2020-09-13 14:57:53 UTC
Is there a workaround?  4 months later it's still an issue

Comment 12 RHEL Program Management 2021-11-11 07:27:10 UTC
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.