Bug 336461 - Metadata file does not match checksum
Metadata file does not match checksum
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: yum (Show other bugs)
7
All Linux
low Severity medium
: ---
: ---
Assigned To: Jeremy Katz
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2007-10-17 13:46 EDT by cje
Modified: 2014-01-21 17:59 EST (History)
7 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2008-03-12 10:59:40 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description cje 2007-10-17 13:46:08 EDT
i know this is a painful one but i can't find an existing bug for it.  this has
been going on for years - there must be a better way of dealing with it!

for the statistically minded, below there's five and a half minutes spent
downloading 16 copies of a 4.7M file (that's 75.2M!) because of this annoying
checksum error.

can someone give a clear explanation of what this is, why it happens, why it
can't be avoided, why there isn't a better way to deal with it and why the
resulting messages aren't more instructive?  that'd ease my annoyance somewhat.

i've marked this "medium" rather than "low" because it's extremely annoying and
confusing and affects new users.  (on IRC people seldom recommend using pup.)

:-)

primary.sqlite.bz2        100% |=========================| 4.7 MB    00:15     
http://www.mirrorservice.org/sites/download.fedora.redhat.com/pub/fedora/linux/releases/7/Everything/x86_64/os/repodata/primary.sqlite.bz2:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.sqlite.bz2        100% |=========================| 4.7 MB    00:21     
http://mirror.karneval.cz/pub/linux/fedora/linux/releases/7/Everything/x86_64/os/repodata/primary.sqlite.bz2:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.sqlite.bz2        100% |=========================| 4.7 MB    00:15     
http://ftp.SURFnet.nl/pub/os/Linux/distr/fedora/linux/releases/7/Everything/x86_64/os/repodata/primary.sqlite.bz2:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.sqlite.bz2        100% |=========================| 4.7 MB    00:16     
ftp://ftp.proxad.net/mirrors/fedora.redhat.com/fedora/linux/releases/7/Everything/x86_64/os/repodata/primary.sqlite.bz2:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.sqlite.bz2        100% |=========================| 4.7 MB    00:20     
http://ftp-stud.hs-esslingen.de/pub/fedora/linux/releases/7/Everything/x86_64/os/repodata/primary.sqlite.bz2:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.sqlite.bz2        100% |=========================| 4.7 MB    00:15     
ftp://ftp.free.fr/mirrors/fedora.redhat.com/fedora/linux/releases/7/Everything/x86_64/os/repodata/primary.sqlite.bz2:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.sqlite.bz2        100% |=========================| 4.7 MB    00:16     
ftp://alviss.et.tudelft.nl/pub/fedora/linux/releases/7/Everything/x86_64/os/repodata/primary.sqlite.bz2:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.sqlite.bz2        100% |=========================| 4.7 MB    00:16     
http://ftp.nluug.nl/pub/os/Linux/distr/fedora/linux/releases/7/Everything/x86_64/os/repodata/primary.sqlite.bz2:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.sqlite.bz2        100% |=========================| 4.7 MB    00:28     
http://ftp.pwr.wroc.pl/pub/linux/fedora/linux/releases/7/Everything/x86_64/os/repodata/primary.sqlite.bz2:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.sqlite.bz2        100% |=========================| 4.7 MB    00:26     
http://sunsite.icm.edu.pl/pub/Linux/fedora/linux/releases/7/Everything/x86_64/os/repodata/primary.sqlite.bz2:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.sqlite.bz2        100% |=========================| 4.7 MB    00:25     
http://mirror.atrpms.net/fedora/linux/releases/7/Everything/x86_64/os/repodata/primary.sqlite.bz2:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.sqlite.bz2        100% |=========================| 4.7 MB    00:18     
ftp://ftp.tudelft.nl/pub/Linux/download.fedora.redhat.com/linux/releases/7/Everything/x86_64/os/repodata/primary.sqlite.bz2:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.sqlite.bz2        100% |=========================| 4.7 MB    00:17     
http://mirrors.ircam.fr/pub/fedora/linux/releases/7/Everything/x86_64/os/repodata/primary.sqlite.bz2:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.sqlite.bz2        100% |=========================| 4.7 MB    00:46     
http://ftp.lip6.fr/ftp/pub/linux/distributions/fedora/releases/7/Everything/x86_64/os/repodata/primary.sqlite.bz2:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
ftp://ftp.ciril.fr/pub/linux/fedora/linux/releases/7/Everything/x86_64/os/repodata/primary.sqlite.bz2:
[Errno 4] IOError: [Errno ftp error] 530 Sorry, the maximum number of clients
(200) from your class are already connected.
Trying other mirror.
primary.sqlite.bz2        100% |=========================| 4.7 MB    00:18     
ftp://ftp.cru.fr/pub/linux/fedora/releases/7/Everything/x86_64/os/repodata/primary.sqlite.bz2:
[Errno -1] Metadata file does not match checksum
Trying other mirror.
primary.sqlite.bz2        100% |=========================| 4.7 MB    00:22     
primary.sqlite.bz2        100% |=========================| 2.2 MB    00:07   

grrrrr.
Comment 1 Seth Vidal 2007-10-17 14:01:38 EDT
Here's how yum does what it does:
for each repo it:
1. grabs a repomd.xml file
2. it grabs the primary.sqlite file

it checks the primary.sqlite file against the repomd.xml file. If the results
don't match it decides that the version from that mirror is out of date and goes
to look at the next mirror for primary.sqlite. 

Occasionally, you get the repomd.xml from a very out of date or very very new
source and then none of the other mirrors match. So you end up running through
the mirrors until you find a match or you run out of mirrors.

The best solution is to run: yum clean metadata

then try the original command again. Occasionally, the reality is the mirrors
just don't match up (even with themselves if they're in the middle of a sync)
and you have to hop mirrors to find a match.

An  idea for solving it:
 - keep track of how many failures we have for the primary file. If it is > than
some number then invalidate the repomd.xml and refetch it.
 

the problem is that idea could end up making you download a lot more data if, in
fact, it is an actual out-of-sync set of mirrors or a broken repository.


suggestions?
Comment 2 James Antill 2007-10-17 14:56:14 EDT
 So I don't think we can/should solve it for the initial fetches.

 BUT I think for updates it'd be better if we did:

1. "download" repomd.xml to tmp file.
2. "download" primary.sql to tmp file.
3. If check fails, delete both files and pretend the current version is the
latest (possibly with a warning being printed). If the check passed rename both
files to be the primary files.

Comment 3 cje 2007-10-17 15:35:20 EDT
it sounds like it's possible that yum doesn't ensure it's getting the repomd.xml
file and the primary.sql file from the same mirror.  is that true?  if so,
wouldn't it be better if yum did do that?  maybe it does by default but some yum
plugins change that behaviour?  (i'm looking at you, fastestmirrorplugin!)

then perhaps we could also try to persuade the mirrors to ensure that those
files only ever appear as a matched pair - then yum could just try once more on
the same mirror and if that fails then drop both files and move on.

if the same thing happens at a second mirror then yum should probably report a
"software repository problem" and suggest trying again tomorrow.  it could also
suggest asking or listening on #fedora IRC to see if others are having the same
problem.

in any case, if you're going to throw away and re-download one of the two files,
how about throwing away the 4k file rather than the 5M one?  i suppose that's
the wrong thing to do if there's a chance it's a dodgy download - but how often
does that happen around the world these days?  still, if we're only going to
re-try once then it's probably okay getting the big file.

finally (well, first of all really) let's change:
[Errno -1] Metadata file does not match checksum
to:
This software repository is in the middle of being updated.
Comment 4 Till Maas 2007-11-08 16:53:40 EST
How about adding a timestamp to the filename of the metadate, this way yum can
search for a distinct filename and the checksum should only not match when the
file is corrupted.
Comment 5 Seth Vidal 2007-11-08 17:18:26 EST
That makes rsyncing w/o --delete ugly and in general is just a kludge.
Comment 6 Till Maas 2007-11-09 03:39:29 EST
(In reply to comment #5)
> That makes rsyncing w/o --delete ugly and in general is just a kludge.

Why is it a kludge? It only makes clear which in which version the metadata
files are, the rpm packages in a repository also contain their version in their
filename.
Comment 7 cje 2007-11-09 07:42:40 EST
hmm.  it does feel a little bit kludgy.  do you mean that the timestamps (or it
could be a version number) in the filenames of those two files would have to
match before yum will try downloading them.

i guess it feels kludgy because it shouldn't be necessary.  in particular - if
you want a timestamp i guess you could use the actual timestamp on the files? 
the packages have different names because multiple versions can be in the same repo.

also, the filenames wouldn't solve the problem on its own - just make it happen
faster as it looks through all the mirrors for the right file name (or a
matching pair) rather than the right file checksum.  that's not what i'm after.

given the passions that surround this whole issue (risk of flame wars etc) i
think we need to work out a very well thought through plan.

on the other hand, i'd appreciate some responses to my suggestions/thoughts in
comment #3.  what do you (all) think about that?
Comment 8 Seth Vidal 2008-03-12 10:59:40 EDT
Okay. A lot of work has gone into yum post 3.2.10 that makes this behavior much
less likely to occur. I'm going to close this as fixed in rawhide but if you can
test it would be appreciated.

Additionally, createrepo has been changed to support unique filenames for the
metadata.
Comment 9 Carl Bingel 2008-04-09 04:45:12 EDT
I have this problem but have noted one thing that may be of help. At work, we
have a http-proxy. If I specify the proxy in /etc/yum.conf, I get the above
described problem about metadata not matching the checksum. But if I remove the
proxy configuration from yum.conf and instead set the http_proxy environment
variable manually and then run yum, everything works fine.

Note You need to log in before you can comment on or make changes to this bug.