1656445 – awscli is missing botocore dependency

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1656445 - awscli is missing botocore dependency

Summary: awscli is missing botocore dependency

Keywords:
Status:	CLOSED CURRENTRELEASE
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	awscli
Sub Component:
Version:	7.5
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	---
Assignee:	Oyvind Albrigtsen
QA Contact:	Brandon Perkins
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2018-12-05 14:20 UTC by Jon Dufresne
Modified:	2019-01-04 15:01 UTC (History)
CC List:	8 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2019-01-04 15:01:48 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Jon Dufresne 2018-12-05 14:20:32 UTC

Description of problem:

After install awscli, usage fails as botocore is not installed.


Version-Release number of selected component (if applicable):

awscli-1.14.28-5.el7_5.1.noarch


How reproducible: 100%


Steps to Reproduce:

# yum install awscli
...
===================================================================================================================================================================================================================
 Package                                                            Arch                                  Version                                                        Repository                           Size
===================================================================================================================================================================================================================
Installing:
 awscli                                                             noarch                                1.14.28-5.el7_5.1                                              base                                1.0 M
Installing for dependencies:
 PyYAML                                                             x86_64                                3.10-11.el7                                                    base                                153 k
 jbigkit-libs                                                       x86_64                                2.0-11.el7                                                     base                                 46 k
 libjpeg-turbo                                                      x86_64                                1.2.90-6.el7                                                   base                                134 k
 libtiff                                                            x86_64                                4.0.3-27.el7_3                                                 base                                170 k
 libwebp                                                            x86_64                                0.3.0-7.el7                                                    base                                170 k
 libyaml                                                            x86_64                                0.1.4-11.el7_0                                                 base                                 55 k
 python-backports                                                   x86_64                                1.0-8.el7                                                      base                                5.8 k
 python-backports-ssl_match_hostname                                noarch                                3.5.0.1-1.el7                                                  base                                 13 k
 python-cffi                                                        x86_64                                1.6.0-5.el7                                                    base                                218 k
 python-dateutil                                                    noarch                                1.5-7.el7                                                      base                                 85 k
 python-docutils                                                    noarch                                0.11-0.3.20130715svn7687.el7                                   base                                1.5 M
 python-enum34                                                      noarch                                1.0.4-1.el7                                                    base                                 52 k
 python-idna                                                        noarch                                2.4-1.el7                                                      base                                 94 k
 python-ipaddress                                                   noarch                                1.0.16-2.el7                                                   base                                 34 k
 python-pillow                                                      x86_64                                2.0.0-19.gitd1c6db8.el7                                        base                                438 k
 python-ply                                                         noarch                                3.4-11.el7                                                     base                                123 k
 python-pycparser                                                   noarch                                2.14-1.el7                                                     base                                104 k
 python-s3transfer                                                  noarch                                0.1.13-1.el7                                                   base                                2.9 M
 python-setuptools                                                  noarch                                0.9.8-7.el7                                                    base                                397 k
 python-six                                                         noarch                                1.9.0-2.el7                                                    base                                 29 k
 python2-cryptography                                               x86_64                                1.7.2-2.el7                                                    base                                502 k
 python2-pyasn1                                                     noarch                                0.1.9-7.el7                                                    base                                100 k

...

# aws
Traceback (most recent call last):
  File "/usr/bin/aws", line 19, in <module>
    import awscli.clidriver
  File "/usr/lib/python2.7/site-packages/awscli/clidriver.py", line 18, in <module>
    import botocore.session
ImportError: No module named botocore.session


Actual results:

ImportError: No module named botocore.session

Expected results:

aws command works

Comment 2 Cole Towsley 2018-12-06 15:23:00 UTC

Jon,

Our development team hasn't been able to easily replicate the issue you've been experiencing wherein awscli is missing botocore on install. To further troubleshoot the circumstances of this failure, it would be helpful if you could open a support ticket directly with Red Hat via the customer portal at access.redhat.com so that our support team can work with you to define the parameters of the issue, nail down the replication, and bring it back to the development team for further analysis. The development team doesn't always have the time or resources to do deep dive troubleshooting for these sorts of issues, so we depend on our support team to handle this in advance. If you have any questions about opening a support ticket or need assistance doign so, please contact Red Hat Support:

https://access.redhat.com/support/contact/technicalSupport

When you open the ticket, feel free to reference this bugzilla in the ticket as well! Thank you!

Regards,

Cole Towsley, RHCE
Software Maintenance Engineer
Red Hat Inc.

Comment 3 Irving Popovetsky 2018-12-10 17:49:02 UTC

Yeah, it appears that the dependency list for the awscli rpm is missing a few things:

[root@ip-10-0-4-107 log]# yum deplist awscli
...
package: awscli.noarch 1.14.28-5.el7_5.1
  dependency: /usr/bin/python2
   provider: python.x86_64 2.7.5-76.el7
  dependency: PyYAML >= 3.10
   provider: PyYAML.x86_64 3.10-11.el7
  dependency: python(abi) = 2.7
   provider: python.x86_64 2.7.5-76.el7
  dependency: python-cryptography >= 1.7.2
   provider: python2-cryptography.x86_64 1.7.2-2.el7
  dependency: python-docutils >= 0.10
   provider: python-docutils.noarch 0.11-0.3.20130715svn7687.el7
  dependency: python-s3transfer >= 0.1.9
   provider: python2-s3transfer.noarch 0.1.10-1.el7
   provider: python-s3transfer.noarch 0.1.13-1.el7


even after installing the python2-botocore package, it appears to be out of sync with the available botocore package according to https://github.com/aws/aws-cli/issues/2999

# aws --version
Traceback (most recent call last):
  File "/bin/aws", line 19, in <module>
    import awscli.clidriver
  File "/usr/lib/python2.7/site-packages/awscli/clidriver.py", line 25, in <module>
    from botocore.history import get_global_history_recorder
ImportError: No module named history

python2-botocore-1.6.0-1.el7.noarch

Comment 4 Christophe Osuna 2018-12-11 05:12:38 UTC

I can reproduce the issue with Vagrant and the following Vagrantfile:

  Vagrant.configure("2") do |config|
    config.vm.box = "bento/centos-7.5"
    config.vm.provision "shell", inline: <<-SHELL
      yum upgrade
      yum install -y awscli
      aws
    SHELL
  end

It seems that no package provides the required file:

  $ yum whatprovides /usr/lib/python2.7/site-packages/boto/session.py
  default: No matches found

The package python-boto3 provides the file boto3/session.py, but this is inconsistent with the source code shipped with awscli.

The package mentioned in the previous comment is from the EPEL repository (while awscli is from base), which would explain why installing python2-botocore introduces further issues.

Comment 5 Jon Dufresne 2018-12-11 14:18:19 UTC

I think there is a typo the 'whatprovides' command. Given the error 'ImportError: No module named botocore.session', I think the 'boto' should be 'botocore'. The point remains the same though, that file doesn't exist.

$ yum whatprovides /usr/lib/python2.7/site-packages/botocore/session.py
...
No matches found

Comment 6 Xavier Bachelot 2018-12-12 14:50:35 UTC

I'm investigating this as we're hitting the issue as well.

In order to get a working awscli package, one either need to downgrade to the version EPEL once shipped :
"""
yum remove awscli python-s3transfer
yum install awscli-1.11.133-1.el7 python2-s3transfer-0.1.10-1.el7 --exclude=python-s3transfer
"""
I'm not sure the packages are still available in EPEL, I'm taking them from a local mirror, but they are likely still available somewhere, at least in Koji.

Alternatively, and this is probably a better fix, one can use an updated python2-botocore package from EPEL on a fully updated RHEL/CentOS 7.6, that is with awscli-1.14.28-5.el7_5.1.noarch and 
python-s3transfer-0.1.13-1.el7.noarch

Here's how I proceeded.
I took current epel7 git branch as the starting point: commit 0088212 (Update to 1.6.0)
Then, rebased it to commit 9cf8648 (Bumping version to 1.10.43 Updates bz#1531330)
Then, updated the EL7 patch to apply against 1.10.43.
The resulting package, python2-botocore-1.10.43-1.el7.1.noarch, when added on top of awscli and python-s3transfer from RHEL/CentOS 7.6, fixes awscli for me.

Here's the PR against python-botocore epel7 branch:
https://src.fedoraproject.org/rpms/python-botocore/pull-request/2

If this is accepted, this would somewhat fix RHEL's awscli package.
This is still not a proper solution to the issue at hand, as RHEL's awscli package should have the python2-botocore dependency added and then python-botocore should also be "adopted" in RHEL.

Comment 8 Dom Gallagher 2018-12-13 13:55:40 UTC

Per https://bugs.centos.org/view.php?id=15518, there appears to be a path inconsistency.
We're using the hack detailed in that report, namely:

sed -i -e 's/\/lib\//\/lib64\//' /usr/lib/python2.7/site-packages/awscli/clidriver.py


Updating (using pip) also resolves this issue, but it bumps the version on a lot of dependencies.

Comment 9 Jon Dufresne 2019-01-02 19:16:47 UTC

I think this issue may have been fixed. After upgrading all system packages, I can no longer reproduce. The change may have come from package python-s3transfer-0.1.13-1.el7.0.1.noarch.

Comment 10 Christophe Osuna 2019-01-04 14:59:39 UTC

(In reply to Jon Dufresne from comment #9)
> I think this issue may have been fixed. After upgrading all system packages,
> I can no longer reproduce. The change may have come from package
> python-s3transfer-0.1.13-1.el7.0.1.noarch.

Same here. With the Vagrantfile proposed in a previous message, there is no longer dependency issue. I agree that the ticket can be closed now.

Comment 11 Oyvind Albrigtsen 2019-01-04 15:01:48 UTC

Thank you for the feedback.

Note You need to log in before you can comment on or make changes to this bug.