RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1233329 - timedhosts corruption: all yum operations fail with ValueError
Summary: timedhosts corruption: all yum operations fail with ValueError
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: python-urlgrabber
Version: 7.1
Hardware: All
OS: Linux
unspecified
urgent
Target Milestone: rc
: ---
Assignee: Valentina Mukhamedzhanova
QA Contact: Martin Frodl
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2015-06-18 16:07 UTC by Zack Cerza
Modified: 2023-09-14 03:00 UTC (History)
7 users (show)

Fixed In Version: python-urlgrabber-3.10-7.el7
Doc Type: Bug Fix
Doc Text:
Cause: running yum when timedhosts file is malformed Consequence: yum crashes Fix: errors while parsing timedhosts file are handled gracefully Result: yum works as expected
Clone Of:
Environment:
Last Closed: 2015-11-19 12:10:28 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:2084 0 normal SHIPPED_LIVE python-urlgrabber bug fix update 2015-11-19 10:49:30 UTC

Description Zack Cerza 2015-06-18 16:07:16 UTC 
Description of problem:
This is greatly affecting Ceph testing.

Somehow, our test machines are getting corrupt timedhosts files. Here is an example of one:
$ cat /var/cache/yum/x86_64/7Server/timedhosts
762:df681ef0b981435a9e8907aa92b46bc71cbd0571
499
NetworkManager
x86_64
1
1.0.0
14.git20150121.b4ea599c.el7
1
/bin/sh
Red_Hat_Enterprise_Li


Version-Release number of selected component (if applicable):
yum-3.4.3-125.el7.noarch

How reproducible:
Unsure what corrupts the timedhosts file, but once it is corrupted, yum is entirely broken

Steps to Reproduce:
1. Notice that all tests are failing with yum errors
2. Attempt to do anything with yum

Actual results:
2015-06-17T21:58:28.938 INFO:teuthology.orchestra.run.magna023:Running: 'sudo yum install -y kernel'
Traceback (most recent call last):
  File "/bin/yum", line 29, in <module>
    yummain.user_main(sys.argv[1:], exit_code=True)
  File "/usr/share/yum-cli/yummain.py", line 367, in user_main
    errcode = main(args)
  File "/usr/share/yum-cli/yummain.py", line 174, in main
    result, resultmsgs = base.doCommands()
  File "/usr/share/yum-cli/cli.py", line 571, in doCommands
    return self.yum_cli_commands[self.basecmd].doCommand(self, self.basecmd, self.extcmds)
  File "/usr/share/yum-cli/yumcommands.py", line 440, in doCommand
    return base.installPkgs(extcmds, basecmd=basecmd)
  File "/usr/share/yum-cli/cli.py", line 967, in installPkgs
    txmbrs = self.install(pattern=arg)
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 4764, in install
    mypkgs = self.pkgSack.returnPackages(patterns=pats,
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 1079, in <lambda>
    pkgSack = property(fget=lambda self: self._getSacks(),
  File "/usr/lib/python2.7/site-packages/yum/__init__.py", line 784, in _getSacks
    self.repos.populateSack(which=repos)
  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 344, in populateSack
    self.doSetup()
  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
    self.retrieveAllMD()
  File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
    dl = repo._async and repo._commonLoadRepoXML(repo)
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1456, in _commonLoadRepoXML
    result = self._getFileRepoXML(local, text)
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1234, in _getFileRepoXML
    size=102400) # setting max size as 100K
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1014, in _getFile
    result = self.grab.urlgrab(misc.to_utf8(relative), local,
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 694, in <lambda>
    grab = property(lambda self: self._getgrab())
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 689, in _getgrab
    self._setupGrab()
  File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 655, in _setupGrab
    failure_callback=mirror_failure)
  File "/usr/lib/python2.7/site-packages/urlgrabber/mirror.py", line 275, in __init__
    self.mirrors.sort(key=estimate, reverse=True)
  File "/usr/lib/python2.7/site-packages/urlgrabber/mirror.py", line 269, in estimate
    speed, fail = _TH.estimate(m['mirror'])
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 2417, in estimate
    _TH.load()
  File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 2366, in load
    host, speed, fail, ts = line.rsplit(' ', 3)
ValueError: need more than 1 value to unpack
Comment 3 Vasu Kulkarni 2015-06-22 17:20:44 UTC 
Can anyone look into this issue? This is causing all test run failures and need the fix ASAP. Thanks
Comment 4 Stanislav Graf 2015-06-22 18:10:12 UTC 
In your logs, first yum command says:
"There are unfinished transactions remaining. You might consider running yum-complete-transaction, or "yum-complete-transaction --cleanup-only" and "yum history redo last", first to finish them. If those don't work you'll have to try removing/installing packages by hand (maybe package-cleanup can help)"

Can you checkout your machine setup?
Comment 6 Zack Cerza 2015-06-23 15:11:17 UTC 
(In reply to Stanislav Graf from comment #4)
> In your logs, first yum command says:
> "There are unfinished transactions remaining. You might consider running
> yum-complete-transaction, or "yum-complete-transaction --cleanup-only" and
> "yum history redo last", first to finish them. If those don't work you'll
> have to try removing/installing packages by hand (maybe package-cleanup can
> help)"
> 
> Can you checkout your machine setup?

Yesterday morning I ran a mass cleanup removing the timedhosts file from all of the test machines; I haven't seen enough jobs scheduled to cause the breakage again. Once I see tht I'll see if what you're suggesting helps. I can tell you that a "yum clean all" does not help.
Comment 7 Zack Cerza 2015-06-23 19:20:14 UTC 
(In reply to Stanislav Graf from comment #4)
> In your logs, first yum command says:
> "There are unfinished transactions remaining. You might consider running
> yum-complete-transaction, or "yum-complete-transaction --cleanup-only" and
> "yum history redo last", first to finish them. If those don't work you'll
> have to try removing/installing packages by hand (maybe package-cleanup can
> help)"
> 
> Can you checkout your machine setup?

Didn't help:
http://pastebin.test.redhat.com/292248
Comment 8 Vasu Kulkarni 2015-06-23 20:54:47 UTC 
Please provide a fix or workaround for this issue, this is causing lots of automated test failure during Ceph Regression runs on RHEL.
Comment 9 Valentina Mukhamedzhanova 2015-06-24 08:04:51 UTC 
(In reply to Zack Cerza from comment #7)
> Didn't help:
> http://pastebin.test.redhat.com/292248

To work around this you can simply delete the /var/cache/yum/x86_64/7Server/timedhosts file.
Comment 14 Dan Mick 2015-07-02 17:19:38 UTC 
Is there a bug open to track the actual fix to yum?  If not, can the yum team please open one and link it to this bug?
Comment 15 Valentina Mukhamedzhanova 2015-07-03 06:43:40 UTC 
(In reply to Dan Mick from comment #14)
> Is there a bug open to track the actual fix to yum?

Could you elaborate on what you mean? The only fix related to this has been made in python-urlgrabber.
Comment 16 Dan Mick 2015-07-06 17:53:39 UTC 
Something is corrupting the timedhosts file; am I correct in assuming we have not yet diagnosed what that thing is?
Comment 17 Valentina Mukhamedzhanova 2015-07-07 08:54:15 UTC 
I don't mind opening a new bug as soon as it is confirmed that it's yum who corrupts the file.

Zack, as far as I understand the problem can be reliably reproduced in your environment. Could you please trace exactly which command makes the file corrupted?
Comment 19 errata-xmlrpc 2015-11-19 12:10:28 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-2084.html
Comment 20 Red Hat Bugzilla 2023-09-14 03:00:53 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days
Note You need to log in before you can comment on or make changes to this bug.