Bug 512454 - attempting to check that existing deltas match signatures does not work
Summary: attempting to check that existing deltas match signatures does not work
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: mash
Version: 12
Hardware: All
OS: Linux
low
medium
Target Milestone: ---
Assignee: Bill Nottingham
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2009-07-17 18:17 UTC by Josh Boyer
Modified: 2014-03-17 03:19 UTC (History)
6 users (show)

Fixed In Version: mash-0.5.20-1.fc13
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2010-10-13 05:58:00 UTC


Attachments (Terms of Use)

Description Josh Boyer 2009-07-17 18:17:51 UTC
Description of problem:

During updates mashing, we call mash with the -p option set to the previous updates directory.  This should allow us to reuse existing drpms in the current push if those particular packages have not been updated.  This is not working for some reason.

Version-Release number of selected component (if applicable):

[jwboyer@releng2 drpms]$ rpm -q createrepo
createrepo-0.9.7-12.el5.noarch


How reproducible:

Seems like always

Steps to Reproduce:
1. Start updates push
2. Watch mash for f11-updates start
3. Swear a lot while it takes forever rebuilding drpms that already exist
  
Actual results:

makedeltarpm called for things that shouldn't be needed

Expected results:

Previous push's drpms reused

Additional info:

mash is called like this from bodhi:

/usr/bin/python /usr/bin/mash -o /mnt/koji/mash/updates/f11-updates-090717.0854 -c /etc/bodhi/mash.conf -f /usr/share/bodhi/comps/comps-f11.xml -p /mnt/koji/mash/updates/f11-updates f11-updates

/mnt/koji/mash/updates/f11-updates is a symlink to the previous repo.  E.g.: 

[jwboyer@releng2 drpms]$ ls -l /mnt/koji/mash/updates/f11-updates
lrwxrwxrwx 1 masher nfsnobody 58 Jul 15 20:09 /mnt/koji/mash/updates/f11-updates -> /mnt/koji/mash/updates/f11-updates-090715.0207/f11-updates

The mash config file specifies two things to delta against.  Namely, GA and the last updates repo:

[f11-updates]
rpm_path = %(arch)s/
source_path = SRPMS/
debuginfo = True
multilib = True
multilib_method = devel
tag = dist-f11-updates
inherit = False
strict_keys = True
keys = D22E77F2
repoviewurl = http://download.fedoraproject.org/pub/fedora/linux/updates/11/%(arch)s/
repoviewtitle = "Fedora 11 Updates - %(arch)s"
arches = i386 x86_64 ppc ppc64
delta = True
delta_dirs = /pub/fedora/linux/releases/11/Everything/%(arch)s/os/,/mnt/koji/mash/updates/f11-updates/%(arch)s/

fpaste showing existing drpms and makedeltarpm running for one of those drpms:
http://fpaste.org/paste/19056

Comment 1 Luke Macken 2009-07-20 13:58:34 UTC
Interesting.  So I'm wondering if this block of code is causing the problem...

    if opts.previous != "":
        if opts.previous[0] != '/':
            opts.previous = os.path.realpath(os.path.join(os.getcwd(), opts.previous))
        for dist in conf.distros:
            dist.previous = opts.previous
            if not dist.delta_dirs:
                dist.delta_dirs = [os.path.join(opts.previous, dist.rpm_path)]


Since we specify the `delta_dirs` in our mash configuration, it seems like the `if not dist.delta_dirs` would never be True, and thus would never get set to the previous path.

CC'ing notting, who can probably provide further insight.

Comment 2 Bill Nottingham 2009-07-20 15:01:16 UTC
No, it sets the 'previous' dir before the 'not dist.delta_dirs' section.

The behavior jwb seemed to be seeing was that it copied the old deltas from the previous dir, and remade them anyway.

Comment 3 Josh Boyer 2009-07-20 15:39:47 UTC
(In reply to comment #2)
> No, it sets the 'previous' dir before the 'not dist.delta_dirs' section.
> 
> The behavior jwb seemed to be seeing was that it copied the old deltas from the
> previous dir, and remade them anyway.  

I can't say either way.  I wasn't able to determine if the drpms were already copied or not.

Comment 4 Josh Boyer 2009-07-20 19:00:13 UTC
Seth added some debugging code to createrepo.  It seems to be showing that mash is not copying the existing drpms as evident here:

/mnt/koji/mash/updates/mash-test/f11-updates/x86_64/drpms/java-1.6.0-openjdk-plugin-1.6.0.0-20.b14.fc11_1.6.0.0-24.b16.fc11.x86_64.drpm does not exist, running makedeltarpm

Current mash working dir:

[jwboyer@releng2 ~]$ ls /mtn/koji/mash/updates/mash-test/f11-updates/x86_64/drpms/java*
ls: /mtn/koji/mash/updates/mash-test/f11-updates/x86_64/drpms/java*: No such file or directory



Directory passed via -p to mash:

[jwboyer@releng2 ~]$ ls /mnt/koji/mash/updates/f11-updates/x86_64/drpms/java*
/mnt/koji/mash/updates/f11-updates/x86_64/drpms/java-1.6.0-openjdk-1.6.0.0-20.b14.fc11_1.6.0.0-24.b16.fc11.x86_64.drpm

So the -p path had the drpm, but the current mash dir does not.  Either it didn't try to copy, or it failed to copy the existing drpms.

Comment 5 James Antill 2009-07-20 20:01:29 UTC
the mnt vs. mtn thing is a typo in the BZ only, yeh?

Comment 6 James Antill 2009-07-20 20:02:44 UTC
Also can you run md5sum on both the old and new .drpm, of above?

Comment 7 Josh Boyer 2009-07-20 22:04:56 UTC
(In reply to comment #5)
> the mnt vs. mtn thing is a typo in the BZ only, yeh?  

No, but createrepo itself complained about it missing and further testing via notting has shown that mash does not like something about the existing rpms.

With a patch from notting, this output was produced from mash:

Checking /mnt/koji/mash/updates/f11-updates-090718.1110/f11-updates/x86_64/drpms/empathy-devel-2.26.1-1.fc11_2.26.2-1.fc11.i586.drpm - does it match a file in our tree?
Looking for empathy-devel-2.26.2-1.fc11.i586.rpm...  found it!
Checking sig... failed

Bill is hrming about this now.

Comment 8 Bill Nottingham 2009-07-21 01:42:34 UTC
We check two things before we copy in a delta rpm for the prior tree:

1) That the NVRE that the delta is for actually matches a package in the current tree. (This should be obvious.)

2) That the signature on the package when the delta was made matches the signature still on the package (in case the package is re-signed)

The second one is done by reading the header of the new rpm (via rpm-python and yum) and the header of the delta rpm, and comparing the signatures.

However, if we look at any delta now, including ones from an unsigned -> signed package (so there *has* to be a signature diff included in the delta)... rpmlib and therefore rpm-python show the delta as having no signature info.

So, the comparison then fails for any new package that's signed. (Which is why rawhide more or less works, and updates always fails.)

It's unclear why this is happening; I'm fairly sure this worked when the code was added to check the sigs, and the delta, when applied to an unsigned package, creates a signed one - so obviously it has the sig info somewhere.

Comment 9 Bill Nottingham 2009-07-21 17:45:30 UTC
Not sure how this ever worked before. The delta carries its own signature; the actual signature of the new RPM is towards the end of the gzipped delta payload.

Options:
1) wait until the check in mash is fixed. This could take a while (unless someone else wants to chip in)
2) turn off the signature check in mash. Updates should work; rawhide will work before packages are signed, but will need to 'clear' the deltas once packages are signed
3) ...?

Comment 10 Luke Macken 2009-07-22 18:29:22 UTC
Regarding option #2, we rely on this mechanism for catching unsigned packages that slip into the mash due to problems with the signing scripts, which still seems to happen every now and then.  The signers can say for sure, but this may be risky to disable, unless we can add some signature checking verification to bodhi's post-mash sanity checking code.

Comment 11 Bill Nottingham 2009-07-22 18:34:16 UTC
That feature is still there; it's just the bit that checks that the deltas match a particular signature on a package that was disabled.

Comment 12 Jonathan Dieter 2009-07-28 16:39:03 UTC
Check out https://fedorahosted.org/koji/attachment/ticket/38/make_delta_from_sig.py

It's ugly code, but it's designed to add/change the package's signature in a deltarpm.  It should be easy to modify it to grab the package's signature.

Comment 13 Bug Zapper 2009-11-16 11:00:55 UTC
This bug appears to have been reported against 'rawhide' during the Fedora 12 development cycle.
Changing version to '12'.

More information and reason for this action is here:
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 14 Fedora Update System 2010-09-28 16:17:39 UTC
mash-0.5.20-1.el5 has been submitted as an update for Fedora EPEL 5.
https://admin.fedoraproject.org/updates/mash-0.5.20-1.el5

Comment 15 Fedora Update System 2010-09-28 16:21:40 UTC
mash-0.5.20-1.fc14 has been submitted as an update for Fedora 14.
https://admin.fedoraproject.org/updates/mash-0.5.20-1.fc14

Comment 16 Fedora Update System 2010-09-28 16:34:03 UTC
mash-0.5.20-1.fc13 has been submitted as an update for Fedora 13.
https://admin.fedoraproject.org/updates/mash-0.5.20-1.fc13

Comment 17 Fedora Update System 2010-09-28 17:31:32 UTC
mash-0.5.20-1.fc14 has been pushed to the Fedora 14 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update mash'.  You can provide feedback for this update here: https://admin.fedoraproject.org/updates/mash-0.5.20-1.fc14

Comment 18 Fedora Update System 2010-10-13 05:56:01 UTC
mash-0.5.20-1.el5 has been pushed to the Fedora EPEL 5 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 19 Fedora Update System 2010-10-13 12:48:48 UTC
mash-0.5.20-1.fc14 has been pushed to the Fedora 14 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 20 Fedora Update System 2011-04-14 20:58:42 UTC
mash-0.5.20-1.fc13 has been pushed to the Fedora 13 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.