Bug 1576985

/usr/lib/python2.7/site-packages/awscli-1.14.28-py2.7.egg-info/requires.txt from this awscli RPM shows: botocore==1.8.32

I found the centos SPEC file here:
https://git.centos.org/blob/rpms!awscli.git/c7/SPECS!awscli.spec

and there's a comment at line 38:
# python-botocore bundled in python-s3transfer
#Requires:       python-botocore >= %{botocore_version}

looking at the s3transfer package, I see no evidence of this:

# rpm -ql python2-s3transfer | grep -c boto
0

OHHHHHH... python-s3transfer versus python2-s3transfer

so, workaround:

# yum erase python2-s3transfer
... also erases awscli ...
# yum install awscli
... also installs python-s3transfer ...
# aws
usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:

  aws help
  aws <command> help
  aws <command> <subcommand> help
aws: error: too few arguments

Still experiencing failures, example:

# aws s3 cp foo s3://mybucket/foo
__init__() got an unexpected keyword argument 'max_bandwidth'

Worked around with:

$ diff -u  transferconfig.py.orig /usr/lib/python2.7/site-packages/awscli/customizations/s3/transferconfig.py
--- transferconfig.py.orig	2018-05-11 15:18:07.210978058 -0400
+++ /usr/lib/python2.7/site-packages/awscli/customizations/s3/transferconfig.py	2018-05-11 15:16:57.475659289 -0400
@@ -109,7 +109,7 @@
         'max_queue_size': 'max_request_queue_size',
         'multipart_threshold': 'multipart_threshold',
         'multipart_chunksize': 'multipart_chunksize',
-        'max_bandwidth': 'max_bandwidth',
+        # 'max_bandwidth': 'max_bandwidth',
     }
     kwargs = {}
     for key, value in runtime_config.items():

Andy,
The packages from EPEL provide the python-* capabilities in addition to the RPM sub-package name (python2-*).  This is really the cause of the whole mess and perhaps the EPEL packages need to alter their provided their capabilities now that RHEL is shipping the same packages.

I came across the issues we're seeing here in a Koji build environment.  Because of how Koji merges repos together and having the base distro repos at a higher priority than EPEL (rightfully so), the packages from the base distro were the only ones available and end up getting installed.  This results in VM images being produced by Koji with broken AWS support.

Koji aside though, any existing systems that have awscli installed will hit the same issue when it is updated and its dependencies really aren't fulfilled at runtime.  Luckily I hit this on my CentOS systems and not my production RHEL systems that are simply subscribed to the HA/RS repos for RHEL.

I created this ticket before diving into the problems too deeply, but later realized there wasn't going to be a good solution because of exactly the items you have come across throughout the various modules.

After looking at possible solutions yesterday, spinning newer versions internally of awscli, python-boto3, python-botocore, and python-s3transfer seemed like the only proper way to solve this.  Rebasing to awscli-1.14.50, python-botocore-1.9.23, python-boto3-1.6.23 and python-s3transfer-0.1.13 was painless and I'd be happy to supply the updated SRPMS if someone wants to run with getting the packages in EPEL updated.  But for me, the issue is resolved until RHEL or EPEL rebase to newer versions than I just listed.

https://github.com/aws/aws-cli/issues/2999 may be related.

"yum swap python2-s3transfer python-s3transfer" seems to help.

Anssi, "helps" in that the aws command doesn't immediately fail, but once you start trying to do things like copy files, there are problems (see comment #4).  The version of s3transfer packaged right now is python-s3transfer-0.1.10-8.el7.noarch

Here is the requires.txt for the version of awscli packaged:

# cat /usr/lib/python2.7/site-packages/awscli-1.14.28-py2.7.egg-info/requires.txt 
botocore==1.8.32
colorama>=0.2.5,<=0.3.7
docutils>=0.10
rsa>=3.1.2,<=3.5.0
s3transfer>=0.1.12,<0.2.0
PyYAML>=3.10,<=3.12

And if you look at the Changelog for s3transfer, it's clear why 0.1.12 is needed:
https://github.com/boto/s3transfer/blob/develop/CHANGELOG.rst#0112

"enhancement:max_bandwidth: Add ability to set maximum bandwidth consumption for streaming of S3 uploads and downloads"

relevant commit (although this shouldn't be cherry-picked on its own):
https://github.com/boto/s3transfer/commit/61e387e361fe20c323f430970a22301906c90e8f

That S3 error sounds like an entirely different issue, which might need its own bug. Also, it would be good to test that on a fresh install without any EPEL packages.

Okay, that's fine.  I've split into https://bugzilla.redhat.com/show_bug.cgi?id=1578083

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3123