Created attachment 527239 [details] yum log Opening this as a yum bug, but maybe it's really a problem with the fedora infrastructure... This morning I went to yum update my workstation. As is often the case, it fell down completely (log attached). The main error seems to be that I get this sort of message repeatedly: ftp://ftp.gtlib.gatech.edu/pub/fedora.redhat/linux/releases/15/Everything/x86_64/debug/repodata/6ad3ddc0b0eb5b314f41c263f9625a9528ed699d8159bebdec56203ab736ccfe-filelists.sqlite.bz2: [Errno 14] FTP Error 550 : ftp://ftp.gtlib.gatech.edu/pub/fedora.redhat/linux/releases/15/Everything/x86_64/debug/repodata/6ad3ddc0b0eb5b314f41c263f9625a9528ed699d8159bebdec56203ab736ccfe-filelists.sqlite.bz2 Trying other mirror. ...the URL changes as it cycles through all the mirrors, but I'm unable to fetch any updates as that file isn't present on any of them. Maybe this is caused by some sort of fan-out lag when pushing to the mirrors, but it's a significant annoyance. Isn't there some more graceful way to handle this? IOW, the signature prefix on that file shouldn't be offered until the mirrors are ready to serve it, right? I don't ever recall having this sort of problem with apt...
> fedora-chromium | 3.4 kB 00:00 .. so we grab a fresh repomd.xml > fedora-chromium/primary_db | 9.9 kB 00:00 .. primary metadata were there > fedora-chromium/filelists_db => 404 .. but not the filelist! Hmm, this suggests there's something wrong with mirrors themselves. I've already seen that few mirrors 404 on primary_db, right after frequent repomd.xml updates, but never all of them. > IOW, the signature prefix on that file shouldn't be offered until the mirrors are ready to serve it, right? When repo gets updated, repomd and MD files are very likely pushed independently, otherwise a single dead mirror would block the whole process. Filelists tend to be large so it may take a while. I'd suggest just try again in few minutes. Just to make sure there's no bug in yum, please run # egrep '(primary|filelists).sqlite' /var/cache/yum/fedora-chromium/repomd.xml # ls -l /var/cache/yum/fedora-chromium/*primary.sqlite and check that both the retrieved *primary.sqlite and the filelist that gave 404 have used the same repomd.xml.
I don't think the problem is with the chromium repo. The log says this at the end: Error: failure: repodata/6ad3ddc0b0eb5b314f41c263f9625a9528ed699d8159bebdec56203ab736ccfe-filelists.sqlite.bz2 from fedora-debuginfo: [Errno 256] No more mirrors to try. I haven't tried to update again since I generated the log, but here's the output you requested: [root@tlielax ~]# egrep '(primary|filelists).sqlite' /var/cache/yum/fedora-chromium/repomd.xml <location href="repodata/2edeb55260fe1e0fc9a00d6baf92bef74976f9393a82e7eb502c342a4bde2d38-primary.sqlite.bz2"/> <location href="repodata/cfdb36c3de1a5ca7d522db63bc2e12bc8c40025beb9cbe8079fca15222b85224-filelists.sqlite.bz2"/> [root@tlielax ~]# ls -l /var/cache/yum/fedora-chromium/*primary.sqlite -rw-r--r--. 1 root root 52224 Oct 10 09:12 /var/cache/yum/fedora-chromium/2edeb55260fe1e0fc9a00d6baf92bef74976f9393a82e7eb502c342a4bde2d38-primary.sqlite ...and here's the same for the fedora-debuginfo repo: # egrep '(primary|filelists).sqlite' /var/cache/yum/fedora-debuginfo/repomd.xml <location href="repodata/1e9f3fb809364d755fa4864aefebf1eaec9b10b0fd6dda330cbe6489075afce9-primary.sqlite.bz2"/> <location href="repodata/6ad3ddc0b0eb5b314f41c263f9625a9528ed699d8159bebdec56203ab736ccfe-filelists.sqlite.bz2"/> [root@tlielax ~]# ls -l /var/cache/yum/fedora-debuginfo/*primary.sqlite-rw-r--r--. 1 root root 9783296 May 21 21:49 /var/cache/yum/fedora-debuginfo/1e9f3fb809364d755fa4864aefebf1eaec9b10b0fd6dda330cbe6489075afce9-primary.sqlite ...I'll leave it to you to tell me whether this is correct or not. Maybe I'm just unlucky when I try to update my machine but this problem happens somewhat frequently, and it often takes quite a while for it to sort itself out (up to an hour in some cases). It really doesn't make any sense to me to ever present a half-baked repo. Wouldn't it be better to sync the data to an alternate spot and then rename the files into place once everything is there?
Right, fedora-chromium is ok. FAIL, didn't read logs properly :/ > repomd.xml: > <location href="repodata/1e9f3fb...-primary.sqlite.bz2"/> > <location href="repodata/6ad3ddc...-filelists.sqlite.bz2"/> > /var/cache/yum/fedora-debuginfo: > 1e9f3fb8...-primary.sqlite > failing url: > ftp://... 6ad3ddc0b0eb5b314f4 ... So the existing 'primary' file and the requested 'filelists' are referenced from the same repomd, that's what I wanted to check, thanks. Now things make more sense.. There's likely nothing wrong with mirrors- it's just yum using stale 'repomd.xml'. That should not happen right after 'yum clean all', maybe we've hit a bug there. I'm pretty sure that removing '/var/cache/yum/fedora-debuginfo/repomd.xml' and running yum again would fix this issue. Before you do that, could you please post 'ls -l' of that directory to see timestamps? Thanks!
Here you go. I've still not tried to update the machine, so let me know if you need more info. $ ls -l /var/cache/yum/fedora-debuginfo/ total 9584 -rw-r--r--. 1 root root 9783296 May 21 21:49 1e9f3fb809364d755fa4864aefebf1eaec9b10b0fd6dda330cbe6489075afce9-primary.sqlite -rw-r--r--. 1 root root 0 Nov 20 2010 395cf51348d27f5af6370e92908b5de9ac606390f589781007babc3c954658ec-filelists.sqlite.bz2 -rw-r--r--. 1 root root 0 Jun 14 15:40 6ad3ddc0b0eb5b314f41c263f9625a9528ed699d8159bebdec56203ab736ccfe-filelists.sqlite.bz2 -rw-r--r--. 1 root root 0 Oct 10 05:52 cachecookie -rw-r--r--. 1 root root 17320 Oct 10 05:52 metalink.xml drwxr-xr-x. 2 root root 4096 May 14 11:18 packages -rw-r--r--. 1 root root 3131 May 18 09:36 repomd.xml
Thank you. Can you please attach your metalink.xml? It's recent, but it likely references invalid repomd.xml. In case you already removed repomd.xml, yum probably complains. Sorry, there's a minor bug in yum, you have to remove 'metalink.xml' or 'cachecookie' instead. Just do that now and yum will retrieve fresh repomd.xml. FYI: mirror 1: http://ftp.linux.ncsu.edu/pub/fedora/linux/releases/15/Everything/i386/debug/repodata/ repomd.xml: 3132 bytes, sha256 5e6f5e16a... mirror 2: http://ftp.nsysu.edu.tw/Fedora/linux/development/15/x86_64/debug/repodata/ repomd.xml: 3131 bytes, sha256 45443ea.. Files on both mirrors are 5 months old, you have used repomd #2 and mirror list #1, but can't figure out why.
5e6f5e16a... is the proper sha256sum for that repomd.xml file, and that is what all of the Fedora app servers are returning to that query, as far as I can tell. However, the ftp.nsysu.edu.tw mirror clearly has the wrong repomd.xml. Normally MirrorManager would recognize this and throw out the mirror. However, I can't find the nsysu.edu.tw mirror in the MirrorManager database anywhere. No URLs by that name, nothing obvious... So I'm not sure how you'd be getting directed there. Also note the URL pasted here points at development/15 which predates the release, and is in fact removed on the master mirrors. Oh look... i386/debug/repodata from ncsu is correct.
Created attachment 527503 [details] fedora-debuginfo/metalink.xml
I don't see anything in the logs that mentions the nsysu.edu.tw URL so I'm not sure that I ever did go there. It's possible I fetched the file from there a while back, but that seems unlikely -- I'm using yum-fastestmirror and I'm on east coast of US -- nowhere near .tw... I'll hold off for a bit on updating the machine in case there's any other info under /var/cache/yum or that would interest you...
Thanks Jeff. That looks like a good metalink for the correct repomd.xml file. You're still getting failures? yum-fastestmirror could well have a bad cache itself - generally it isn't needed as much anymore because MM does a good job of handing out fast mirrors (statistically) preferred over slow mirrors.
Chatting with Jeff via IRC, he had incorrect data in /var/cache/yum/fedora-debuginfo/ (particularly the repomd.xml file, with timestamp newer than what was on the Fedora mirrors). yum clean all was not removing it. By manually removing that directory, yum was able to download the correct repomd.xml file from a mirror, and everything else started working as expected. Unclear why yum clean all doesn't just nuke /var/cache/yum/*, or at least nuke all the repomd.xml files previously downloaded. Also unclear why it wasn't downloading new repomd.xml files. In Jeff's cache: -rw-r--r--. 1 root root 3131 May 18 09:36 /var/cache/yum/fedora-debuginfo/repomd.xml On the mirrors: -rw-rw-r-- 1 ftp ftp 3132 May 17 13:33 repomd.xml Problem fixed for Jeff, so will close, but curious what the underlying failure really is.
Matt, thanks for help! 'yum clean' issue: Repository 'fedora-debug' is disabled by default, but the 'auto-update-debuginfo' plugin enables it. 'yum clean all' cleans only enabled repositories and does not call 'prereposetup' hook, so the plugin does not run beforehand. FIX: Make 'yum clean' call prereposetup hook (but not actual reposetup, since we don't want to download metadata just to remove it)? 'repomd.xml' timestamp: When urlgrabber downloads a file, it's timestamp is set to whatever was in HTTP header. The 'cachecookie' file is used to check if metadata are current. This file is touched even if download/verification of new 'repomd.xml' fails and yum reverts to the old one. This should be changed too, probably. Downloading an older repomd.xml file is fine (reget=None), but yum then calculates ts=max(timestamp) from <timestamp> tags in it, and refuses to use it if new_ts < old_ts. Maybe we should turn off that timestamp check, too? if (self.timestamp_check and old_repo_XML.timestamp > self.repoXML.timestamp): logger.warning("Not using downloaded repomd.xml because it is " "older than what we have:\n" " Current : %s\n Downloaded: %s" %
Had the same issue. Thanks for the info.