Bug 1479533 - [starter-us-east-1] error from yum module during upgrade
[starter-us-east-1] error from yum module during upgrade
Product: OpenShift Container Platform
Classification: Red Hat
Component: Installer (Show other bugs)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: 3.7.0
Assigned To: Luke Meyer
Johnny Liu
Depends On:
  Show dependency treegraph
Reported: 2017-08-08 14:00 EDT by Justin Pierce
Modified: 2017-11-28 17:06 EST (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2017-11-28 17:06:30 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Justin Pierce 2017-08-08 14:00:17 EDT
Description of problem:

Version-Release number of the following components:
openshift-ansible v3.

How reproducible:

Steps to Reproduce:
1. Ran upgrade on large HA cluster (>100 nodes). Occurred on one. 

Actual results:

Using module file /usr/lib/python2.7/site-packages/ansible/modules/core/packaging/os/yum.py
<> SSH: EXEC ssh -o ControlMaster=auto -o ControlPersist=600s -o StrictHostKeyChecking=no -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/home/opsmedic/.ansible/cp/ansible-ssh-%h-%p-%r '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"''
fatal: [starter-us-east-1-node-compute-8bcb6]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "invocation": {
        "module_args": {
            "conf_file": null, 
            "disable_gpg_check": false, 
            "disablerepo": null, 
            "enablerepo": null, 
            "exclude": null, 
            "install_repoquery": true, 
            "list": null, 
            "name": [
            "state": "latest", 
            "update_cache": false, 
            "validate_certs": true
    "msg": "Traceback (most recent call last):\n  File \"/usr/bin/yum\", line 29, in <module>\n    yummain.user_main(sys.argv[1:], exit_code=True)\n  File \"/usr/share/yum-cli/yummain.py\", line 370, in user_main\n    errcode = main(args)\n  File \"/usr/share/yum-cli/yummain.py\", line 179, in main\n    result, resultmsgs = base.doCommands()\n  File \"/usr/share/yum-cli/cli.py\", line 573, in doCommands\n    return self.yum_cli_commands[self.basecmd].doCommand(self, self.basecmd, self.extcmds)\n  File \"/usr/share/yum-cli/yumcommands.py\", line 1626, in doCommand\n    ypl = base.returnPkgLists(extcmds, repoid=repoid)\n  File \"/usr/share/yum-cli/cli.py\", line 1400, in returnPkgLists\n    ignore_case=True, repoid=repoid)\n  File \"/usr/lib/python2.7/site-packages/yum/__init__.py\", line 3005, in doPackageLists\n    for (n,a,e,v,r) in self.up.getUpdatesList():\n  File \"/usr/lib/python2.7/site-packages/yum/__init__.py\", line 1093, in <lambda>\n    up = property(fget=lambda self: self._getUpdates(),\n  File \"/usr/lib/python2.7/site-packages/yum/__init__.py\", line 838, in _getUpdates\n    self._up = rpmUtils.updates.Updates(self.rpmdb.simplePkgList(), self.pkgSack.simplePkgList())\n  File \"/usr/lib/python2.7/site-packages/yum/__init__.py\", line 1074, in <lambda>\n    pkgSack = property(fget=lambda self: self._getSacks(),\n  File \"/usr/lib/python2.7/site-packages/yum/__init__.py\", line 778, in _getSacks\n    self.repos.populateSack(which=repos)\n  File \"/usr/lib/python2.7/site-packages/yum/repos.py\", line 386, in populateSack\n    sack.populate(repo, mdtype, callback, cacheonly)\n  File \"/usr/lib/python2.7/site-packages/yum/yumRepo.py\", line 242, in populate\n    mydbtype)\n  File \"/usr/lib/python2.7/site-packages/yum/yumRepo.py\", line 287, in _check_uncompressed_db_gen\n    cached=repo.cache)\n  File \"/usr/lib/python2.7/site-packages/yum/misc.py\", line 1165, in repo_gen_decompress\n    return decompress(filename, dest=dest, check_timestamps=True)\n  File \"/usr/lib/python2.7/site-packages/yum/misc.py\", line 1152, in decompress\n    os.utime(out, (fi.st_mtime, fi.st_mtime))\nOSError: [Errno 2] No such file or directory: '/var/cache/yum/x86_64/7Server/rhel-7-server-rpms/gen/primary_db.sqlite'\n", 
    "rc": 1, 
    "results": []
Comment 1 Scott Dodson 2017-08-08 14:08:54 EDT
We need to add failure tolerance to all node operations rather than just the drain and upgrade phases.
Comment 2 Scott Dodson 2017-09-25 11:25:03 EDT
We've added retries around yum transactions.
Comment 4 Johnny Liu 2017-09-27 10:58:21 EDT
Already did verification in https://bugzilla.redhat.com/show_bug.cgi?id=1482551#c8, and PASS.
Comment 8 errata-xmlrpc 2017-11-28 17:06:30 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.