Bug 374801 - comps-f8.xml corrupt and so yum-updatesd continually re-downloads it
comps-f8.xml corrupt and so yum-updatesd continually re-downloads it
Status: CLOSED CURRENTRELEASE
Product: Fedora
Classification: Fedora
Component: yum-updatesd (Show other bugs)
8
All Linux
high Severity urgent
: ---
: ---
Assigned To: Jeremy Katz
Fedora Extras Quality Assurance
:
: 374961 375491 376151 376261 376361 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2007-11-10 11:22 EST by Mark
Modified: 2008-01-12 11:01 EST (History)
14 users (show)

See Also:
Fixed In Version: repopushed
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2007-11-12 11:17:37 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
Log of HTTP messages sent by yum-updatesd-helper (2.04 KB, text/plain)
2007-11-10 12:04 EST, Ville-Pekka Vainio
no flags Details
netstat -tp (updatesd connecting to various mirrors) (3.19 KB, text/plain)
2007-11-10 13:42 EST, Mauricio Teixeira
no flags Details

  None (edit)
Description Mark 2007-11-10 11:22:24 EST
yum-updatesd-helper is banging away at the network and won't allow package
manager to run when needed.  I have 2 machines and they both are doing this. 
There are also many posts on the forums about this problem.  The
yum-updatesd-helper process is out of control.  When machines are sitting idle,
there is constant network activity from the process and yum can't be used.


This affects Fedora 8 installs


No steps needed to reproduce
Comment 1 Ville-Pekka Vainio 2007-11-10 12:04:44 EST
Created attachment 253991 [details]
Log of HTTP messages sent by yum-updatesd-helper
Comment 2 Ville-Pekka Vainio 2007-11-10 12:07:34 EST
I'm seeing this too, it makes using yum almost impossible, since the network and
the yum lock are always taken. I've attached some log I made with fireshark, it
seems updatesd-helper constantly downloads updates comps-f8.xml here.
Comment 3 Luigi Tarenga 2007-11-10 12:56:44 EST
i killed the yum-updatesd-helper process and did some experiment.

"yum update" works, i was able to install new packages.

"yum grouplist" does not work, here the output that seems related
to the helper problem:
[root@bug1 updates]# yum grouplist
Setting up Group Process
http://ftp.gui.uva.es/sites/fedora.redhat.com/linux/updates/8/x86_64/repodata/comps-f8.xml:
[Errno 4] IOError: <urlopen error (-3, 'Temporary failure in name resolution')>
Trying other mirror.
comps-f8.xml              100% |=========================| 1.2 MB    00:13     
http://www.jur-linux.org/download/fedora/updates/8/x86_64/repodata/comps-f8.xml:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
comps-f8.xml              100% |=========================| 1.2 MB    00:13     
ftp://ftp.uninett.no/pub/linux/Fedora/updates/8/x86_64/repodata/comps-f8.xml:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
....
the process go on with the same error on many mirror (i didn't try them all :))

Comment 4 Mauricio Teixeira 2007-11-10 13:41:59 EST
+1!

I've seen this happen in my desktop with x86_64 version, but not on my laptop
with i386 version.

Can't tell where does it comes from also, but looking at netstat it seems that
updatesd is trying to get repository files from all the mirrors repeately.
Comment 5 Mauricio Teixeira 2007-11-10 13:42:49 EST
Created attachment 254031 [details]
netstat -tp (updatesd connecting to various mirrors)
Comment 6 Rolf Linder 2007-11-10 15:00:58 EST
have same problem on an i386 fedora8. yum-updatesd does download several hundred
MB's until its dead...

yum update over console is working fine.
Comment 7 Andre Robatino 2007-11-10 15:55:20 EST
> yum update over console is working fine.

Not for me, I get the following indefinitely with command-line yum, which I use
exclusively.  I have to kill the yum-updatesd and yum-updatesd-helper processes
(the latter with -9) before yum will work.

Existing lock /var/run/yum.pid: another copy is running as pid 2552.
Another app is currently holding the yum lock; waiting for it to exit...
Another app is currently holding the yum lock; waiting for it to exit...
Another app is currently holding the yum lock; waiting for it to exit...
Another app is currently holding the yum lock; waiting for it to exit...
Another app is currently holding the yum lock; waiting for it to exit...
Comment 8 Ken 2007-11-10 15:59:02 EST
Same problem here. I did a fresh install of Fedora 8 this morning and when I
went to use Software Updater, it started retrieving software information. It sat
there retrieving software information for 40 minutes and caused 321 MB of
download traffic before I killed the process.

Using Yum in the terminal works fine.
Comment 9 James Antill 2007-11-10 16:06:17 EST
> It sat there retrieving software information for 40 minutes and caused 321 MB of
download traffic before I killed the process.

 This is "normal" in that there were/are 100s of MBs of updates for Fed-8
available as of this morning. So as soon as you reboot into Fed-8 yum-updatesd
will spawn a yum-updates-helper which will start downloading all of those
updates. While it's doing this it takes the yum lock, so you can't run yum/pirut
until it's finished.

 If you "service yum-updatesd stop" that will not kill the yum-updatesd-helper
_because_ if yum-updatesd-helper is in the middle of installing the downloaded
rpms it can do very bad things to your rpmdb. _If_ you know it's still
downloading you can kill yum-updatesd-helper manually, this should let you use
yum/pirut fine (at least I've not seen any evidence of a problem, other than
slow network).

 If someone has a bug that isn't explained by the above, please add a comment.
Comment 10 James Antill 2007-11-10 16:06:46 EST
*** Bug 374961 has been marked as a duplicate of this bug. ***
Comment 11 Andre Robatino 2007-11-10 16:16:47 EST
  I first experienced the problem today AFTER I downloaded all the updates about
1.5 hours ago, and rebooted into the new kernel.  I noticed that the networking
light on my router was just going, and going, and going, and yum wouldn't work
because of the lock error.  In F7 that would happen for a short time but then
stop when yum-updatesd finished checking.  This time, there were no new updates,
so that's not the explanation.  I don't know why it didn't happen after the last
time I booted (about 2 days ago).
Comment 12 Ville-Pekka Vainio 2007-11-10 16:36:10 EST
(In reply to comment #9)
>  If someone has a bug that isn't explained by the above, please add a comment.

This behavior is not explained by available updates. For example in my case I
have installed all the available updates with the yum CLI. But if I start
yum-updatesd, it will still (apparently) download the repodata files constantly,
until I kill yum-updatesd-helper with kill -9.

Comment 13 Andre Robatino 2007-11-10 16:40:35 EST
  To verify that the problem is reproducible, I've just rebooted and logged in
again.  All updates are applied, including last night's batch, and I'm running
the new kernel.  In F7, even with a large batch of new updates, it didn't take
more than a minute or two for it to finish its hourly check and display the
puplet icon.  At this point, it's been running about 15 minutes even with no
updates (I also have a "yum update" running in a terminal, giving the lock error
repeatedly, so will know if it finishes).  I will report back if it finishes
within an hour or so, or kill it if it's still running by then.  I should also
mention that the "yum update" I did to download last night's batch of updates
finished in normal time, so the server can't be that overloaded.
Comment 14 James Antill 2007-11-10 17:39:20 EST
 Ok, Andre/Ville-Pekka can you tell me what options yum-updatesd-helper is
running with? (ps axuwwwwf | fgrep yum-updates)
 Can you try running the following and pasting the output:

/usr/libexec/yum-updatesd-helper --debug <options>

...where <options> are replaced with the options you see from the ps.
 For instance I'm trying:

/usr/libexec/yum-updatesd-helper -c -d --deps --debug

...and while it says there are 6 updates available, it exits almost immediately.

 Also can you run (and see if anything is different):

% rpm -q yum yum-updatesd yum-utils python-urlgrabber python
yum(0:3.2.7-1.fc8).noarch
yum-updatesd(1:0.7-1.fc8).noarch
yum-utils(0:1.1.8-1.fc8).noarch
python-urlgrabber(0:3.0.0-3.fc8).noarch
python(0:2.5.1-15.fc8).x86_64
Comment 15 Andre Robatino 2007-11-10 17:50:22 EST
  Amazingly enough, my yum finally started up just a few minutes ago, letting me
know there are no updates.  Running "yum clean metadata" and then "yum update"
finishes in less than a minute, but yum-updatesd took about an hour, or about 2
orders of magnitude longer.  I can't get yum-updatesd-helper running again
without logging out and back in, but the following appeared in /var/log/messages
at what is probably the time that yum-updatesd-helper exited:

Nov 10 17:39:21 localhost yum-updatesd-helper: error getting update info:
'module' object has no attribute 'GroupError'

  I haven't changed any defaults.  The only customization I've made is
installing yum-presto, and the livna-release, dribble-release, and adobe-release
packages to enable those repos.

[root@localhost ~]# rpm -q yum yum-updatesd yum-utils python-urlgrabber python
yum-3.2.7-1.fc8
yum-updatesd-0.7-1.fc8
yum-utils-1.1.8-1.fc8
python-urlgrabber-3.0.0-3.fc8
python-2.5.1-15.fc8
[root@localhost ~]#

Comment 16 Mauricio Teixeira 2007-11-10 18:24:37 EST
Hey, guys, I found the root cause of the problem.

After facing the same problem with yum-updatesd and pirut, I little of network
debug with 'wireshark' gave me the directions on what should I be looking for.

# yum grouplist
Setting up Group Process
comps-f8.xml              100% |=========================| 1.2 MB    00:06     
http://ftp.nluug.nl/pub/os/Linux/distr/fedora/linux/updates/8/x86_64/repodata/comps-f8.xml:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
comps-f8.xml              100% |=========================| 1.2 MB    00:09     
http://ftp.funet.fi/pub/mirrors/fedora.redhat.com/pub/fedora/linux/updates/8/x86_64/repodata/comps-f8.xml:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
(...)

And it keeps going until all the mirrors are checked.

So, the problem is that the either the file comps-f8.xml is corrupted, or the
information of the file in repomd.xml is wrong. I any case, a rebuild of the
repodata would be a way to fix.
Comment 17 Andre Robatino 2007-11-10 18:30:48 EST
  In case it helps, yum-updatesd-helper is running again, and I get

[andre@localhost ~]$ ps axuwwwwf | fgrep yum-updates
root      2272  0.0  0.4  21564  8476 ?        SN   16:25   0:00 /usr/bin/python
-tt /usr/sbin/yum-updatesd
root      3445  4.5  3.9  92472 82176 ?        SN   18:25   0:05  \_
/usr/bin/python -tt /usr/libexec/yum-updatesd-helper --check --dbus
andre     3526  0.0  0.0   4016   540 pts/1    S+   18:27   0:00          |    
  \_ fgrep yum-updates

but from the previous comment it sounds like the cause is found.  If not I'll
try running

/usr/libexec/yum-updatesd-helper --debug --check --dbus

when the current instance exits.  I'm also currently running "yum update ; date"
to verify that the time the lock is freed is the same time the error message
appears in /var/log/messages.
Comment 18 Andre Robatino 2007-11-10 20:59:37 EST
  I verified that the error in /var/log/messages appears when
yum-updatesd-helper exits.  Running the debug command shows nothing new:

[root@localhost ~]# /usr/libexec/yum-updatesd-helper --debug --check --dbus
Loading "presto" plugin
Setting up and reading Presto delta metadata
No Presto metadata available for livna
No Presto metadata available for adobe-linux-i386
No Presto metadata available for updates
No Presto metadata available for dribble
No Presto metadata available for fedora
Setting up and reading Presto delta metadata
No Presto metadata available for livna
No Presto metadata available for adobe-linux-i386
No Presto metadata available for updates
No Presto metadata available for dribble
No Presto metadata available for fedora
Check for updates failed: 'module' object has no attribute 'GroupError'
[root@localhost ~]#

Everything except the last line is printed in the first few seconds, but the
last line takes about an hour to print.
Comment 19 James Antill 2007-11-11 00:28:15 EST
> And it keeps going until all the mirrors are checked.

 So I'm not seeing anything like this atm. ... that might just mean I've been
lucky, or it might be some weird mirrorlist problem. I'll try and check some
more tomorrow or monday.

 But to mitigate this you could pick a mirror from:
mirrors.fedoraproject.org/publiclist/Fedora/8/ ... and use that as the baseurl
commenting out the mirrorlist. Then even if you're still having problems
yum-updatesd et. al. will only download the xml data once (then they'll error out).
Comment 20 Luigi Tarenga 2007-11-11 06:08:47 EST
hi all,
i think it is just a problem of checksum for the comps-f8.xml.
why i can say that.
i had as all of you, pirut and pup not working. 
looking at the "yum grouplist" error i see the checksum error so i
tryed to "skip" that check. to make so i modifyed
/usr/lib/python2.5/site-packages/yum/yumRepo.py
(it's a TEMPORARY WORKAROUND DO NOT USE IT )

here the diff:
--- yumRepo.py  2007-11-11 11:59:11.000000000 +0100
+++ /usr/lib/python2.5/site-packages/yum/yumRepo.py     2007-11-11
11:59:39.000000000 +0100
@@ -802,10 +802,11 @@
         except Errors.RepoError, e:
             raise URLGrabError(-3, 'Error performing checksum')
 
-        if l_csum == r_csum:
-            return 1
-        else:
-            raise URLGrabError(-1, 'Metadata file does not match checksum')
+        #if l_csum == r_csum:
+        #    return 1
+        #else:
+        #    raise URLGrabError(-1, 'Metadata file does not match checksum')
+        return 1
 
 
 

as you can see from the diff it skip the checksum check and always return
ok (1).
now pirut works fine!!! and pup also, it just give this
output
[root@bug1 ~]# pup
Failed to add groups file for repository: updates - comps file is empty/damaged

but works.
i think the solution is just to correct the checkrum on all mirror.
bye bye
Comment 21 Fabio Airoldi 2007-11-11 09:29:15 EST
I think the problem stays somewhere in the updates repository. Running pirut
--repo=updates does not work (the progress bar stops at 100%).
It works with any other repository (i tested fedora, livna and adobe-linux-i386).
I hope to be helpful..
Bye
Comment 22 James Antill 2007-11-11 10:48:55 EST
 Well I'm seeing it now too, repomd.xml says:

  <data type="group">
    <location href="repodata/comps-f8.xml"/>
    <checksum type="sha">e4e400aa672481bf1d9e70c008e1c95ee98c0cb7</checksum>
    <timestamp>1194644692</timestamp>
  </data>

...(ts = Fri Nov  9 21:44:52 2007 UTC) but my comps-f8.xml is (1194668210)
2007-11-10 04:16 with sha1sum 3a850436d3988e93235903d8a90a5c98f14a4908.
 I altered the repomd.xml file by hand and that gets past the download check,
but the comps file itself is broken, lines 287-289:

    <name xml:lang="bs">Podrška za arapski</name>
    </name>
    <name xml:lang="ko">아랍어 지원</name>

...and line 1331:

    <name xml:lang="el">Î¥Ï<80>οÏ<83>Ï<84>ήÏ<81>ιξη
Bhutanese</office.org-langpack-bn</packagereq>

...and that error seems to be repeated a bunch of times. So it's basically
unfixable by hand, *sigh*.
Comment 23 James Antill 2007-11-11 11:00:23 EST
 It's worth noting that if you just change the repomd.xml file to contain the
"correct" timestamp dna sha1sum pirut/etc. will work for everything that doesn't
need the compas data.
Comment 24 Luke Macken 2007-11-11 11:26:28 EST
The comps-f8.xml/repomd.xml that bodhi mashed matches up correctly.  We hit this
issue once before, and were hoping that we wouldn't hit it again.  Our master
mirror may be corrupting things.

In the mean time, I kicked off other f8-updates mash.  Hopefully this will sync
out uncorrupted.
Comment 25 Ron Yorston 2007-11-11 11:47:29 EST
Getting the comps-f8.xml file fixed would be good, but there's an underlying
problem with yum-updatesd-helper that needs to be fixed too.  Especially since
this sort of corruption has happened before.

yum-updatesd-helper should be more intelligent about its use of bandwidth.
Comment 26 Brian Pepple 2007-11-11 13:34:03 EST
*** Bug 376261 has been marked as a duplicate of this bug. ***
Comment 27 Andre Robatino 2007-11-11 13:55:57 EST
  Please change back the name of this bug, and also increase the severity.  The
misleading name is probably leading to duplicate bug reports.  The problem has
nothing to do with the volume of updates, since it happens even when all updates
are applied.
Comment 28 Joe Christy 2007-11-11 14:01:48 EST
FWIW firefox, when it tries to open:
http://mirror.anl.gov/pub/fedora/linux/updates/8/x86_64/repodata/comps-f8.xml

reports:

XML Parsing Error: mismatched tag. Expected: </group>.
Location:
http://mirror.anl.gov/pub/fedora/linux/updates/8/x86_64/repodata/comps-f8.xml
Line Number 288, Column 7:    </name>
------^

Comment 29 Joe Christy 2007-11-11 14:03:46 EST
Is there some compelling reason to mark this as medium severity?

The effect of this bug is to render pirut, pup, pupplet, and
system-config-packages _unusable_ for _anything_, to any f8 who has the updates
repo enabled.

Any ETA on when this can be fixed?

Comment 30 Luke Macken 2007-11-11 14:10:02 EST
I'm seeing a good comps-f8.xml on our mirror, as well as populated on a couple
of others.  Can someone verify that this is fixed?
Comment 31 Mauricio Teixeira 2007-11-11 15:06:04 EST
A few mirrors are still out of sync, but at least now it's taking less tries to
get the right file. Thanks.
Comment 32 James Antill 2007-11-12 09:44:27 EST
*** Bug 375491 has been marked as a duplicate of this bug. ***
Comment 33 Jeremy Katz 2007-11-12 11:17:37 EST
Fixed a typo in yum-updatesd that made the impact a little bit worse than it
would have otherwise been also and will push a new yum-updatesd within the next
few days or week.  Given that we shouldn't hit this when the repo is in the
correct state, I'm not going to rush the update out today and instead going to
make sure there's nothing else which needs fixing
Comment 34 Jeremy Katz 2007-11-12 11:26:40 EST
*** Bug 376151 has been marked as a duplicate of this bug. ***
Comment 35 Jeremy Katz 2007-11-12 11:27:00 EST
*** Bug 376361 has been marked as a duplicate of this bug. ***
Comment 36 Andre Robatino 2007-11-12 14:15:23 EST
  It would still be nice for historical purposes at least, so people can look up
this bug, if the summary for it was changed to something at least remotely
accurate (in particular, it has nothing to do with updates).
Comment 37 Radek Bíba 2007-11-15 06:50:52 EST
I'm seeing this problem with comps-f8.xml again now.
Comment 38 James Antill 2007-11-15 09:03:43 EST
 I've just checked:

http://download.fedora.redhat.com/pub/fedora/linux/updates/8/x86_64/repodata/

...and both the sha1sum and xmllint is happy, please note that this sometimes
happens a little bit due to mirrors not syncing immediately.
Comment 39 Alexei Podtelezhnikov 2007-11-15 11:23:16 EST
It was a bad idea to push, pull, and never push back firefox update... some 
mirrors are still broken because of that, so yum and yum-updatesd keeps 
stumbling there.

Comment 40 Radek Bíba 2007-11-15 11:38:42 EST
Okay, it's alright here now as well, where 'it' means the comps file.
yum-updatesd-helper (or whatever is responsible for the madness) is still
broken, though...
Comment 41 Ville-Pekka Vainio 2007-11-27 10:48:01 EST
I'm seeing this yet again. Could you please fix yum-updatesd so that it won't
constantly download even if the xml files are broken? Also, I think this bug
should be reopened until the issue is really fixed.
Comment 42 NILMONI DEB 2007-11-27 11:36:00 EST
This is getting really annoying. This bug is preventing users from fixing 
other bugs like https://bugzilla.redhat.com/show_bug.cgi?id=368871#c10 . I 
tried to update a bug-fixed package from command-line ->

$ su -c 'yum --enablerepo=updates-testing update system-config-network' 

Existing lock /var/run/yum.pid: another copy is running as pid 2996.
Another app is currently holding the yum lock; waiting for it to exit...
Another app is currently holding the yum lock; waiting for it to exit...
Another app is currently holding the yum lock; waiting for it to exit...

"ps -ef" shows /usr/libexec/yum-updatesd-helper running in the background.

The title of this bug needs changing and this bug really needs fixing.
Comment 43 Ron Yorston 2007-12-06 04:12:44 EST
(In reply to comment #33)
> Fixed a typo in yum-updatesd that made the impact a little bit worse than it
> would have otherwise been also and will push a new yum-updatesd within the next
> few days or week.  Given that we shouldn't hit this when the repo is in the
> correct state, I'm not going to rush the update out today and instead going to
> make sure there's nothing else which needs fixing

Jeremy, when can we expect this fix to mitigate the effect of broken mirrors?

Comment 44 Ville-Pekka Vainio 2008-01-12 11:01:54 EST
I'm seeing this yet again. I had one F8 installation that hadn't been used for a
while, after starting it yum-updatesd-helper started the download. The system
has been up now for 3.5 hours and it's still constantly downloading the comps file.

alviss.et.tudelft.nl is the latest mirror it was downloading from, but I guess
that specific mirror has nothing to do with this. When will yum-updatesd be
fixed so that it won't constantly download the file even if the file is broken?

Note You need to log in before you can comment on or make changes to this bug.