Bug 1233329
| Summary: | timedhosts corruption: all yum operations fail with ValueError | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Zack Cerza <zcerza> |
| Component: | python-urlgrabber | Assignee: | Valentina Mukhamedzhanova <vmukhame> |
| Status: | CLOSED ERRATA | QA Contact: | Martin Frodl <mfrodl> |
| Severity: | urgent | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.1 | CC: | isenfeld, james.antill, sgraf, vakulkar, vmukhame, yweinste, zcerza |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | python-urlgrabber-3.10-7.el7 | Doc Type: | Bug Fix |
| Doc Text: |
Cause: running yum when timedhosts file is malformed
Consequence: yum crashes
Fix: errors while parsing timedhosts file are handled gracefully
Result: yum works as expected
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2015-11-19 12:10:28 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Can anyone look into this issue? This is causing all test run failures and need the fix ASAP. Thanks In your logs, first yum command says: "There are unfinished transactions remaining. You might consider running yum-complete-transaction, or "yum-complete-transaction --cleanup-only" and "yum history redo last", first to finish them. If those don't work you'll have to try removing/installing packages by hand (maybe package-cleanup can help)" Can you checkout your machine setup? (In reply to Stanislav Graf from comment #4) > In your logs, first yum command says: > "There are unfinished transactions remaining. You might consider running > yum-complete-transaction, or "yum-complete-transaction --cleanup-only" and > "yum history redo last", first to finish them. If those don't work you'll > have to try removing/installing packages by hand (maybe package-cleanup can > help)" > > Can you checkout your machine setup? Yesterday morning I ran a mass cleanup removing the timedhosts file from all of the test machines; I haven't seen enough jobs scheduled to cause the breakage again. Once I see tht I'll see if what you're suggesting helps. I can tell you that a "yum clean all" does not help. (In reply to Stanislav Graf from comment #4) > In your logs, first yum command says: > "There are unfinished transactions remaining. You might consider running > yum-complete-transaction, or "yum-complete-transaction --cleanup-only" and > "yum history redo last", first to finish them. If those don't work you'll > have to try removing/installing packages by hand (maybe package-cleanup can > help)" > > Can you checkout your machine setup? Didn't help: http://pastebin.test.redhat.com/292248 Please provide a fix or workaround for this issue, this is causing lots of automated test failure during Ceph Regression runs on RHEL. (In reply to Zack Cerza from comment #7) > Didn't help: > http://pastebin.test.redhat.com/292248 To work around this you can simply delete the /var/cache/yum/x86_64/7Server/timedhosts file. Is there a bug open to track the actual fix to yum? If not, can the yum team please open one and link it to this bug? (In reply to Dan Mick from comment #14) > Is there a bug open to track the actual fix to yum? Could you elaborate on what you mean? The only fix related to this has been made in python-urlgrabber. Something is corrupting the timedhosts file; am I correct in assuming we have not yet diagnosed what that thing is? I don't mind opening a new bug as soon as it is confirmed that it's yum who corrupts the file. Zack, as far as I understand the problem can be reliably reproduced in your environment. Could you please trace exactly which command makes the file corrupted? Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHBA-2015-2084.html The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |
Description of problem: This is greatly affecting Ceph testing. Somehow, our test machines are getting corrupt timedhosts files. Here is an example of one: $ cat /var/cache/yum/x86_64/7Server/timedhosts 762:df681ef0b981435a9e8907aa92b46bc71cbd0571 499 NetworkManager x86_64 1 1.0.0 14.git20150121.b4ea599c.el7 1 /bin/sh Red_Hat_Enterprise_Li Version-Release number of selected component (if applicable): yum-3.4.3-125.el7.noarch How reproducible: Unsure what corrupts the timedhosts file, but once it is corrupted, yum is entirely broken Steps to Reproduce: 1. Notice that all tests are failing with yum errors 2. Attempt to do anything with yum Actual results: 2015-06-17T21:58:28.938 INFO:teuthology.orchestra.run.magna023:Running: 'sudo yum install -y kernel' Traceback (most recent call last): File "/bin/yum", line 29, in <module> yummain.user_main(sys.argv[1:], exit_code=True) File "/usr/share/yum-cli/yummain.py", line 367, in user_main errcode = main(args) File "/usr/share/yum-cli/yummain.py", line 174, in main result, resultmsgs = base.doCommands() File "/usr/share/yum-cli/cli.py", line 571, in doCommands return self.yum_cli_commands[self.basecmd].doCommand(self, self.basecmd, self.extcmds) File "/usr/share/yum-cli/yumcommands.py", line 440, in doCommand return base.installPkgs(extcmds, basecmd=basecmd) File "/usr/share/yum-cli/cli.py", line 967, in installPkgs txmbrs = self.install(pattern=arg) File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 4764, in install mypkgs = self.pkgSack.returnPackages(patterns=pats, File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1079, in <lambda> pkgSack = property(fget=lambda self: self._getSacks(), File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 784, in _getSacks self.repos.populateSack(which=repos) File "/usr/lib/python2.7/site-packages/yum/repos.py", line 344, in populateSack self.doSetup() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup self.retrieveAllMD() File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD dl = repo._async and repo._commonLoadRepoXML(repo) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1456, in _commonLoadRepoXML result = self._getFileRepoXML(local, text) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1234, in _getFileRepoXML size=102400) # setting max size as 100K File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1014, in _getFile result = self.grab.urlgrab(misc.to_utf8(relative), local, File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 694, in <lambda> grab = property(lambda self: self._getgrab()) File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 689, in _getgrab self._setupGrab() File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 655, in _setupGrab failure_callback=mirror_failure) File "/usr/lib/python2.7/site-packages/urlgrabber/mirror.py", line 275, in __init__ self.mirrors.sort(key=estimate, reverse=True) File "/usr/lib/python2.7/site-packages/urlgrabber/mirror.py", line 269, in estimate speed, fail = _TH.estimate(m['mirror']) File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 2417, in estimate _TH.load() File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 2366, in load host, speed, fail, ts = line.rsplit(' ', 3) ValueError: need more than 1 value to unpack