Bug 1576985 - awscli.clidriver is unable to import history module
Summary: awscli.clidriver is unable to import history module
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: python-s3transfer
Version: 7.5
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: rc
: ---
Assignee: Oyvind Albrigtsen
QA Contact: Brandon Perkins
URL:
Whiteboard:
Depends On:
Blocks: 1581142
TreeView+ depends on / blocked
 
Reported: 2018-05-10 22:10 UTC by Aron Parsons
Modified: 2018-10-30 10:13 UTC (History)
8 users (show)

Fixed In Version: python-s3transfer-0.1.13-1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1581142 (view as bug list)
Environment:
Last Closed: 2018-10-30 10:12:40 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:3123 0 None None None 2018-10-30 10:13:01 UTC

Description Aron Parsons 2018-05-10 22:10:58 UTC
Description of problem:
awscli-1.14.28-5.el7.noarch is unusable.  This wouldn't be a big deal, except that this package now trumps the version from EPEL that most people rely on.


Version-Release number of selected component (if applicable):
awscli-1.14.28-5.el7.noarch


How reproducible:
always


Steps to Reproduce:
1. yum -y install awscli
2. aws


Actual results:
Traceback (most recent call last):
  File "/usr/bin/aws", line 19, in <module>
    import awscli.clidriver
  File "/usr/lib/python2.7/site-packages/awscli/clidriver.py", line 25, in <module>
    from botocore.history import get_global_history_recorder


Expected results:
usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:

  aws help
  aws <command> help
  aws <command> <subcommand> help
aws: error: too few arguments


Additional info:

Comment 2 Andy Grimm 2018-05-11 19:07:03 UTC
/usr/lib/python2.7/site-packages/awscli-1.14.28-py2.7.egg-info/requires.txt from this awscli RPM shows: botocore==1.8.32

I found the centos SPEC file here:
https://git.centos.org/blob/rpms!awscli.git/c7/SPECS!awscli.spec

and there's a comment at line 38:
# python-botocore bundled in python-s3transfer
#Requires:       python-botocore >= %{botocore_version}

looking at the s3transfer package, I see no evidence of this:

# rpm -ql python2-s3transfer | grep -c boto
0

Comment 3 Andy Grimm 2018-05-11 19:11:21 UTC
OHHHHHH... python-s3transfer versus python2-s3transfer

so, workaround:

# yum erase python2-s3transfer
... also erases awscli ...
# yum install awscli
... also installs python-s3transfer ...
# aws
usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:

  aws help
  aws <command> help
  aws <command> <subcommand> help
aws: error: too few arguments

Comment 4 Andy Grimm 2018-05-11 19:18:53 UTC
Still experiencing failures, example:

# aws s3 cp foo s3://mybucket/foo
__init__() got an unexpected keyword argument 'max_bandwidth'

Worked around with:

$ diff -u  transferconfig.py.orig /usr/lib/python2.7/site-packages/awscli/customizations/s3/transferconfig.py
--- transferconfig.py.orig	2018-05-11 15:18:07.210978058 -0400
+++ /usr/lib/python2.7/site-packages/awscli/customizations/s3/transferconfig.py	2018-05-11 15:16:57.475659289 -0400
@@ -109,7 +109,7 @@
         'max_queue_size': 'max_request_queue_size',
         'multipart_threshold': 'multipart_threshold',
         'multipart_chunksize': 'multipart_chunksize',
-        'max_bandwidth': 'max_bandwidth',
+        # 'max_bandwidth': 'max_bandwidth',
     }
     kwargs = {}
     for key, value in runtime_config.items():

Comment 5 Aron Parsons 2018-05-11 19:55:45 UTC
Andy,
The packages from EPEL provide the python-* capabilities in addition to the RPM sub-package name (python2-*).  This is really the cause of the whole mess and perhaps the EPEL packages need to alter their provided their capabilities now that RHEL is shipping the same packages.

I came across the issues we're seeing here in a Koji build environment.  Because of how Koji merges repos together and having the base distro repos at a higher priority than EPEL (rightfully so), the packages from the base distro were the only ones available and end up getting installed.  This results in VM images being produced by Koji with broken AWS support.

Koji aside though, any existing systems that have awscli installed will hit the same issue when it is updated and its dependencies really aren't fulfilled at runtime.  Luckily I hit this on my CentOS systems and not my production RHEL systems that are simply subscribed to the HA/RS repos for RHEL.

I created this ticket before diving into the problems too deeply, but later realized there wasn't going to be a good solution because of exactly the items you have come across throughout the various modules.

After looking at possible solutions yesterday, spinning newer versions internally of awscli, python-boto3, python-botocore, and python-s3transfer seemed like the only proper way to solve this.  Rebasing to awscli-1.14.50, python-botocore-1.9.23, python-boto3-1.6.23 and python-s3transfer-0.1.13 was painless and I'd be happy to supply the updated SRPMS if someone wants to run with getting the packages in EPEL updated.  But for me, the issue is resolved until RHEL or EPEL rebase to newer versions than I just listed.

Comment 6 Anssi Johansson 2018-05-12 14:48:29 UTC
https://github.com/aws/aws-cli/issues/2999 may be related.

Comment 7 Anssi Johansson 2018-05-14 17:27:16 UTC
"yum swap python2-s3transfer python-s3transfer" seems to help.

Comment 8 Andy Grimm 2018-05-14 17:56:27 UTC
Anssi, "helps" in that the aws command doesn't immediately fail, but once you start trying to do things like copy files, there are problems (see comment #4).  The version of s3transfer packaged right now is python-s3transfer-0.1.10-8.el7.noarch

Here is the requires.txt for the version of awscli packaged:

# cat /usr/lib/python2.7/site-packages/awscli-1.14.28-py2.7.egg-info/requires.txt 
botocore==1.8.32
colorama>=0.2.5,<=0.3.7
docutils>=0.10
rsa>=3.1.2,<=3.5.0
s3transfer>=0.1.12,<0.2.0
PyYAML>=3.10,<=3.12

And if you look at the Changelog for s3transfer, it's clear why 0.1.12 is needed:
https://github.com/boto/s3transfer/blob/develop/CHANGELOG.rst#0112

"enhancement:max_bandwidth: Add ability to set maximum bandwidth consumption for streaming of S3 uploads and downloads"

relevant commit (although this shouldn't be cherry-picked on its own):
https://github.com/boto/s3transfer/commit/61e387e361fe20c323f430970a22301906c90e8f

Comment 9 Anssi Johansson 2018-05-14 18:12:22 UTC
That S3 error sounds like an entirely different issue, which might need its own bug. Also, it would be good to test that on a fresh install without any EPEL packages.

Comment 10 Andy Grimm 2018-05-14 18:51:38 UTC
Okay, that's fine.  I've split into https://bugzilla.redhat.com/show_bug.cgi?id=1578083

Comment 17 errata-xmlrpc 2018-10-30 10:12:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3123


Note You need to log in before you can comment on or make changes to this bug.