Description of problem: During updates mashing, we call mash with the -p option set to the previous updates directory. This should allow us to reuse existing drpms in the current push if those particular packages have not been updated. This is not working for some reason. Version-Release number of selected component (if applicable): [jwboyer@releng2 drpms]$ rpm -q createrepo createrepo-0.9.7-12.el5.noarch How reproducible: Seems like always Steps to Reproduce: 1. Start updates push 2. Watch mash for f11-updates start 3. Swear a lot while it takes forever rebuilding drpms that already exist Actual results: makedeltarpm called for things that shouldn't be needed Expected results: Previous push's drpms reused Additional info: mash is called like this from bodhi: /usr/bin/python /usr/bin/mash -o /mnt/koji/mash/updates/f11-updates-090717.0854 -c /etc/bodhi/mash.conf -f /usr/share/bodhi/comps/comps-f11.xml -p /mnt/koji/mash/updates/f11-updates f11-updates /mnt/koji/mash/updates/f11-updates is a symlink to the previous repo. E.g.: [jwboyer@releng2 drpms]$ ls -l /mnt/koji/mash/updates/f11-updates lrwxrwxrwx 1 masher nfsnobody 58 Jul 15 20:09 /mnt/koji/mash/updates/f11-updates -> /mnt/koji/mash/updates/f11-updates-090715.0207/f11-updates The mash config file specifies two things to delta against. Namely, GA and the last updates repo: [f11-updates] rpm_path = %(arch)s/ source_path = SRPMS/ debuginfo = True multilib = True multilib_method = devel tag = dist-f11-updates inherit = False strict_keys = True keys = D22E77F2 repoviewurl = http://download.fedoraproject.org/pub/fedora/linux/updates/11/%(arch)s/ repoviewtitle = "Fedora 11 Updates - %(arch)s" arches = i386 x86_64 ppc ppc64 delta = True delta_dirs = /pub/fedora/linux/releases/11/Everything/%(arch)s/os/,/mnt/koji/mash/updates/f11-updates/%(arch)s/ fpaste showing existing drpms and makedeltarpm running for one of those drpms: http://fpaste.org/paste/19056
Interesting. So I'm wondering if this block of code is causing the problem... if opts.previous != "": if opts.previous[0] != '/': opts.previous = os.path.realpath(os.path.join(os.getcwd(), opts.previous)) for dist in conf.distros: dist.previous = opts.previous if not dist.delta_dirs: dist.delta_dirs = [os.path.join(opts.previous, dist.rpm_path)] Since we specify the `delta_dirs` in our mash configuration, it seems like the `if not dist.delta_dirs` would never be True, and thus would never get set to the previous path. CC'ing notting, who can probably provide further insight.
No, it sets the 'previous' dir before the 'not dist.delta_dirs' section. The behavior jwb seemed to be seeing was that it copied the old deltas from the previous dir, and remade them anyway.
(In reply to comment #2) > No, it sets the 'previous' dir before the 'not dist.delta_dirs' section. > > The behavior jwb seemed to be seeing was that it copied the old deltas from the > previous dir, and remade them anyway. I can't say either way. I wasn't able to determine if the drpms were already copied or not.
Seth added some debugging code to createrepo. It seems to be showing that mash is not copying the existing drpms as evident here: /mnt/koji/mash/updates/mash-test/f11-updates/x86_64/drpms/java-1.6.0-openjdk-plugin-1.6.0.0-20.b14.fc11_1.6.0.0-24.b16.fc11.x86_64.drpm does not exist, running makedeltarpm Current mash working dir: [jwboyer@releng2 ~]$ ls /mtn/koji/mash/updates/mash-test/f11-updates/x86_64/drpms/java* ls: /mtn/koji/mash/updates/mash-test/f11-updates/x86_64/drpms/java*: No such file or directory Directory passed via -p to mash: [jwboyer@releng2 ~]$ ls /mnt/koji/mash/updates/f11-updates/x86_64/drpms/java* /mnt/koji/mash/updates/f11-updates/x86_64/drpms/java-1.6.0-openjdk-1.6.0.0-20.b14.fc11_1.6.0.0-24.b16.fc11.x86_64.drpm So the -p path had the drpm, but the current mash dir does not. Either it didn't try to copy, or it failed to copy the existing drpms.
the mnt vs. mtn thing is a typo in the BZ only, yeh?
Also can you run md5sum on both the old and new .drpm, of above?
(In reply to comment #5) > the mnt vs. mtn thing is a typo in the BZ only, yeh? No, but createrepo itself complained about it missing and further testing via notting has shown that mash does not like something about the existing rpms. With a patch from notting, this output was produced from mash: Checking /mnt/koji/mash/updates/f11-updates-090718.1110/f11-updates/x86_64/drpms/empathy-devel-2.26.1-1.fc11_2.26.2-1.fc11.i586.drpm - does it match a file in our tree? Looking for empathy-devel-2.26.2-1.fc11.i586.rpm... found it! Checking sig... failed Bill is hrming about this now.
We check two things before we copy in a delta rpm for the prior tree: 1) That the NVRE that the delta is for actually matches a package in the current tree. (This should be obvious.) 2) That the signature on the package when the delta was made matches the signature still on the package (in case the package is re-signed) The second one is done by reading the header of the new rpm (via rpm-python and yum) and the header of the delta rpm, and comparing the signatures. However, if we look at any delta now, including ones from an unsigned -> signed package (so there *has* to be a signature diff included in the delta)... rpmlib and therefore rpm-python show the delta as having no signature info. So, the comparison then fails for any new package that's signed. (Which is why rawhide more or less works, and updates always fails.) It's unclear why this is happening; I'm fairly sure this worked when the code was added to check the sigs, and the delta, when applied to an unsigned package, creates a signed one - so obviously it has the sig info somewhere.
Not sure how this ever worked before. The delta carries its own signature; the actual signature of the new RPM is towards the end of the gzipped delta payload. Options: 1) wait until the check in mash is fixed. This could take a while (unless someone else wants to chip in) 2) turn off the signature check in mash. Updates should work; rawhide will work before packages are signed, but will need to 'clear' the deltas once packages are signed 3) ...?
Regarding option #2, we rely on this mechanism for catching unsigned packages that slip into the mash due to problems with the signing scripts, which still seems to happen every now and then. The signers can say for sure, but this may be risky to disable, unless we can add some signature checking verification to bodhi's post-mash sanity checking code.
That feature is still there; it's just the bit that checks that the deltas match a particular signature on a package that was disabled.
Check out https://fedorahosted.org/koji/attachment/ticket/38/make_delta_from_sig.py It's ugly code, but it's designed to add/change the package's signature in a deltarpm. It should be easy to modify it to grab the package's signature.
This bug appears to have been reported against 'rawhide' during the Fedora 12 development cycle. Changing version to '12'. More information and reason for this action is here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
mash-0.5.20-1.el5 has been submitted as an update for Fedora EPEL 5. https://admin.fedoraproject.org/updates/mash-0.5.20-1.el5
mash-0.5.20-1.fc14 has been submitted as an update for Fedora 14. https://admin.fedoraproject.org/updates/mash-0.5.20-1.fc14
mash-0.5.20-1.fc13 has been submitted as an update for Fedora 13. https://admin.fedoraproject.org/updates/mash-0.5.20-1.fc13
mash-0.5.20-1.fc14 has been pushed to the Fedora 14 testing repository. If problems still persist, please make note of it in this bug report. If you want to test the update, you can install it with su -c 'yum --enablerepo=updates-testing update mash'. You can provide feedback for this update here: https://admin.fedoraproject.org/updates/mash-0.5.20-1.fc14
mash-0.5.20-1.el5 has been pushed to the Fedora EPEL 5 stable repository. If problems still persist, please make note of it in this bug report.
mash-0.5.20-1.fc14 has been pushed to the Fedora 14 stable repository. If problems still persist, please make note of it in this bug report.
mash-0.5.20-1.fc13 has been pushed to the Fedora 13 stable repository. If problems still persist, please make note of it in this bug report.