Bug 188986 - "fastestmirror" plugin makes checksums errors
Summary: "fastestmirror" plugin makes checksums errors
Keywords:
Status: CLOSED UPSTREAM
Alias: None
Product: Fedora
Classification: Fedora
Component: yum
Version: 5
Hardware: i386
OS: Linux
medium
medium
Target Milestone: ---
Assignee: Luke Macken
QA Contact:
URL:
Whiteboard:
: 189463 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2006-04-14 08:58 UTC by Thomas Canniot
Modified: 2016-09-20 02:36 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2006-10-14 03:39:05 UTC
Type: ---
Embargoed:


Attachments (Terms of Use)

Description Thomas Canniot 2006-04-14 08:58:49 UTC
Description of problem: When "fastestmirror" plugin is installed, and only when
it is installed, yum can't resolve checksum on different officiel repositories
(extras, core, updates) 


Version-Release number of selected component (if applicable):
yum-fastestmirror-0.5-1
yum-utils-0.5-1
yum-2.6.0-1


How reproducible:
always

Steps to Reproduce:
1. Install yum-fastestmirror-0.5-1
2. launch yum
3.
  
Actual results:
primary.xml.gz            100% |=========================| 923 kB    00:06
http://download.fedora.redhat.com/pub/fedora/linux/extras/5/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================|    0 B    00:30
ftp://fedora.mirrors.tds.net/pub/fedora-core-extras/5/i386/repodata/primary.xml.gz:
[Errno 4] Socket Error: timed out
Trying other mirror.
primary.xml.gz            100% |=========================| 923 kB    00:45
http://mirror.hiwaay.net/redhat/fedora/linux/extras/5/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 920 kB    00:04
http://www.gtlib.cc.gatech.edu/pub/fedora.redhat/linux/extras/5/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum

... and this until it has checked lots and lots of repo

Expected results:
Should match checksums

Additional info: When yum-fastestmirror-0.5-1 is not installed, everything is fine

Comment 1 Luke Macken 2006-04-24 04:43:49 UTC
I haven't been able to reproduce this at all.

The only thing fastestmirror does is sort the mirrorlist, so I'm very confused
as to how it would be causing checksum issues.

Seth, any ideas ?

Comment 2 Luke Macken 2006-04-24 04:45:48 UTC
*** Bug 189463 has been marked as a duplicate of this bug. ***

Comment 3 Seth Vidal 2006-04-24 04:52:01 UTC
The only idea I can come up with is that this happened as a result of the
fastest mirror for the bug reporter being in a deeply broken state.

I'd say close it worksforme unless the reporter can recreate it.


Comment 4 Luke Macken 2006-04-24 14:18:00 UTC
Thomas, are you still experiencing this issue?  Are you able to recreate it?

Comment 5 Chitlesh GOORAH 2006-04-24 14:39:56 UTC
Chitlesh's here:) (from a duplicate)
I'm having the same issue again and again

-bash-3.1# yum install koffice
Loading "fastestmirror" plugin
Loading "installonlyn" plugin
Setting up Install Process
Setting up repositories
kde                                                                  [1/7]
kde                       100% |=========================|  951 B    00:00
macromedia                                                           [2/7]
macromedia                100% |=========================|  951 B    00:00
kde-all                                                              [3/7]
kde-all                   100% |=========================|  951 B    00:00
extras                                                               [4/7]
extras                    100% |=========================| 1.1 kB    00:00
updates                                                              [5/7]
updates                   100% |=========================|  951 B    00:00
core                                                                 [6/7]
core                      100% |=========================| 1.1 kB    00:00
livna                                                                [7/7]
livna                     100% |=========================|  951 B    00:00
Loading mirror speeds from cached hostfile
Reading repository metadata in from local files
primary.xml.gz            100% |=========================| 1.0 MB    00:04
ftp://fedora.bu.edu/fedora/extras/5/i386/repodata/primary.xml.gz: [Errno -1]
Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 958 kB    00:04
ftp://fedora.mirrors.tds.net/pub/fedora-core-extras/5/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 787 kB    00:05
ftp://mirror.newnanutilities.org/pub/fedora/linux/extras/5/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 1.0 MB    00:05
http://download.fedora.redhat.com/pub/fedora/linux/extras/5/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.xml.gz            100% |=========================| 937 kB    00:15
extras    : ################################################## 2620/2620
Added 26 new packages, deleted 85 old in 6.38 seconds
Parsing package install arguments
No Match for argument: koffice
Nothing to do
[1]+  Done                    kyum
-bash-3.1#  

Comment 6 Luke Macken 2006-04-24 23:43:56 UTC
Sorry, I didn't mean to close this bug :)

This issue is probably fastestmirrors doing..

Since the plugin sorts the mirrorlist -after- the repomd gets pulled, then if
either the repomd or the faster mirror are out of sync -- then *boom*.

Moving fastestmirror from the postreposetup conduit to the prereposetup won't
work, since right after the prerepo callback, yum runs setup()->baseurlSetup(),
which will re-populate the mirror list. 

The first fix I can think of off the top of my head would be to hack
basurlSetup() to check if the attribute 'urls' already exists, and then return.

Seth, what do you think ?

Comment 7 Thomas Canniot 2006-04-25 06:53:33 UTC
No i'm not able to predict if the behaviour will happen or not. I only use yum
in command line and it seems that sometimes it works fine and sometimes
everything crashes. I can't say when or why.
All I can say is that it seems to crash far less often *without* the plugin.

Comment 8 Luke Macken 2006-04-27 19:54:19 UTC
For the past few days I've been experiencing this issue quite frequently, with a
clean cache and --no-plugins.

Comment 9 Kasper Dupont 2006-04-30 12:11:32 UTC
I see the same symptoms on FC4 without yum-fastestmiror installed.

Comment 10 Michal Jaegermann 2006-05-12 03:09:22 UTC
Yes, this is easy to reproduce in recent times.  I strongly doubt that
yum-fastestmiror has anything to do with it as primary.xml.gz which eventually
is accepted at least _may_ be of a different size than whatever was tried
on failed attempts.

It seems to me that 'extras', both for FC5 and for development, are 
particularly, although not exclusively, prone to this issue. See
https://www.redhat.com/archives/fedora-test-list/2006-May/msg00137.html
for an example.  I would think that metadata and checksums are indeed
out of sync but I have no idea why.

With yum-fastestmiror not there one is simply trying mirrors in a different
order and that may strongly affect an outcome.

Comment 11 Phil Schaffner 2006-07-01 11:06:39 UTC
Have also been seeing this behavior with FC5 for some time now.  It is highly
variable, sometimes occurring many times and others very few, occasionally not
at all.

Comment 12 Brian Kosick 2006-07-04 22:02:00 UTC
I have this problem also, and I believe that there is not enough info to
recreate this problem. I can recreate it by following the following steps:

Install the yum-fastestmirror rpm using yum
yum install yum-fastestmirror

This breaks yum casuing the checksum error, I noticed that the filesize
indicator is now twice the size of the file download (467kb vs 282kb)

yum update
Loading "installonlyn" plugin
Loading "fastestmirror" plugin
Setting up Update Process
Setting up repositories
atrpms                                                               [1/6]
atrpms                    100% |=========================|  951 B    00:00
core                                                                 [2/6]
core                      100% |=========================| 1.1 kB    00:00
updates                                                              [3/6]
updates                   100% |=========================|  951 B    00:00
freshrpms                                                            [4/6]
freshrpms                 100% |=========================|  951 B    00:00
extras                                                               [5/6]
extras                    100% |=========================|  951 B    00:00
release                                                              [6/6]
release                   100% |=========================|  951 B    00:00
Loading mirror speeds from cached hostfile
Reading repository metadata in from local files
primary.xml.gz            100% |=========================| 148 kB    00:05
primary.xml.gz            100% |=========================| 767 kB    00:15
primary.xml.gz            100% |=========================| 467 kB    00:10
http://download.fedora.redhat.com/pub/fedora/linux/core/updates/5/i386/repodata/primary.xml.gz:
[Errno -1] Metadata file does not match checksum
Trying other mirror.

2)  Uninstalling yum-fastestmirror fixes the issue

yum update
yum -y update
Loading "installonlyn" plugin
Setting up Update Process
Setting up repositories
atrpms                                                               [1/6]
core                                                                 [2/6]
updates                                                              [3/6]
freshrpms                                                            [4/6]
extras                                                               [5/6]
release                                                              [6/6]
Reading repository metadata in from local files
primary.xml.gz            100% |=========================| 282 kB    00:07
updates   : ################################################## 1013/1013
Added 31 new packages, deleted 9 old in 2.80 seconds

Brian

Comment 13 Brian Kosick 2006-07-04 22:07:54 UTC
Sorry I forgot the list of installed rpms:
yum-2.6.1-0.fc5
gnome-yum-0.1.3-1.fc5
yum-utils-0.6-2.fc5
yum-fastestmirror-0.6-2.fc5

Comment 14 Hans Ulrich Niedermann 2006-07-09 23:44:19 UTC
I can reproduce this bug without the fastestmirror on my local repository.

I have patched yum a little to actually show the two checksums which do not
match. The result suggests that yum confuses the checksum for primary.xml with
the checksum for primary.xml.gz:

[root@mir ~]# vi +/"Metadata file does not match checksum"
/usr/lib/python2.4/site-packages/yum/repos.py

Change line to:

        raise URLGrabError(-1, 'Metadata file does not match checksum (%s,%s)' %
(repr(l_csum), repr(r_csum)))


Test run (immediately after "yum clean all"):

[root@mir ~]# yum update
Setting up Update Process
Setting up repositories
ndim                                                                 [1/9]
core-source                                                          [2/9]
livna                                                                [3/9]
ndim-source                                                          [4/9]
core                                                                 [5/9]
extras-source                                                        [6/9]
updates                                                              [7/9]
extras                                                               [8/9]
ndim-private                                                         [9/9]
Reading repository metadata in from local files
primary.xml.gz            100% |=========================| 2.8 kB    00:00
http://localhost/%7Euser/fedora/ndim/5/i386/repodata/primary.xml.gz: [Errno -1]
Metadata file does not match checksum
('4feaafb7fc0c4d9bd9a02ad56b933b97d2512f4a','1c468f0c53e624c273b00755a31519819d37e93d')
Trying other mirror.
Error: failure: repodata/primary.xml.gz from ndim: [Errno 256] No more mirrors
to try.
[root@mir ~]#


The real sha1sums from the file are:

4feaafb7fc0c4d9bd9a02ad56b933b97d2512f4a  primary.xml.gz
1c424bef62ec55f2122f64cff12ce1a71ed0170c  primary.xml


The repomd.xml section is:

  <data type="primary">
    <location xml:base="http://localhost/~user/fedora/ndim/5/i386"
href="repodata/primary.xml.gz"/>
    <checksum type="sha">4feaafb7fc0c4d9bd9a02ad56b933b97d2512f4a</checksum>
    <timestamp>1152487623</timestamp>
    <open-checksum
type="sha">1c424bef62ec55f2122f64cff12ce1a71ed0170c</open-checksum>
  </data>


This is FC5 with

  yum-2.6.1
  createrepo-0.4.3


Comment 15 Hans Ulrich Niedermann 2006-07-10 00:20:14 UTC
Argh. Sorry. I overlooked something and arrived at the wrong conclusion. The
r_csum starts with "1c4" like the uncompressed checksum in the example above,
but they continue differently.

So yum is *not* comparing the checksum of the compressed data to the checksum of
the uncompressed data.

However, yum is still comparing garbage for some reason. I know the files are
there, I have checked the primary.xml{,.gz} sha1sums against repomd.xml, the
files are actually delivered by the web server, ...


Comment 16 Hans Ulrich Niedermann 2006-07-10 00:27:58 UTC
Summary of my research so far:

It seems that in the /usr/lib/python2.4/site-packages/yum/repos.py file the
Repository._checkMD(self, fn, mdtype) method somehow calculates garbage for
r_csum, while l_csum is fine.

This means that the function self.repoXML.primaryChecksum() returns garbage,
wherever that may be defined.

Comment 17 Hans Ulrich Niedermann 2006-07-10 11:36:44 UTC
Further examination reveals that yum gets an outdated version of repomd.xml and
a current version of primary.xml. Checking the old checksum vs. the new one will
then fail.

More investigation going on at http://es.lauft.net/yum-checksum-error.html - I
will add another comment here when I have reached the root of the problem.

Comment 18 Hans Ulrich Niedermann 2006-07-10 12:45:45 UTC
It seems that my problem was not exactly the fault of yum but the fault of squid.

The LFUDNA policy as recommended on the
http://fedoraproject.org/wiki/Extras/MockTricks page caches repomd.xml far
longer than reasonable. This results in yum working with a current
primary.xml.gz and an ancient repomd.xml, and thus failing.

The workaround is easy: Unset the proxy= line in /etc/yum.conf. I have no idea
how to properly fix it, though.

And I am also not sure that my problem is the exact same one that was described
by the other people above.

I'd suggest that these people try run "yum clean all" once and then "yum update"
(you can abort with "n"). Then have a look at the dates of the files
/var/cache/yum/REPONAME/ and whether the checksums defined in repomd.xml match
the checksums of the files themselves. That would determine whether their
problem and mine have the same underlying cause or something different.

Comment 19 Michal Jaegermann 2006-07-10 15:39:35 UTC
> The workaround is easy: Unset the proxy= line in /etc/yum.conf.

This may help in some situations, if you can do something like that,
but the problem happens too in an absence of any proxies or caches
which can be controlled on a receiver end.

Quoted output shows that not only checksums but even sizes of fetched
primary.xml.gz files are changing.  It looks like that there are
various places where things may go out of sync and yum has troubles
with that.

Comment 20 Hans Ulrich Niedermann 2006-08-13 18:44:28 UTC
A new hint. My repository has this directory structure:

  fedora
  +- 5
     +- i386
     |  +- repodata
     |     +- primary.xml.gz
     +- noarch
        +- repodata
           +- primary.xml.gz

The /var/cache/yum/foobar/primary.xml.gz only contains packages from i386/, but
none at all from noarch/.

Comment 21 Luke Macken 2006-10-14 03:39:05 UTC
The new mirrorlist cgi script resolves this issue server-side, since it makes
sure all the mirrors it returns are in sync with the master repomd.xml (within 1
hour I believe).

Just make sure you are using the following mirrorlist for your repos (for
rawhide, replace 'core-$releasever' with 'rawhide'):
    
mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=core-$releasever&arch=$basearch


Note You need to log in before you can comment on or make changes to this bug.