Just to note, I've noticed this behavior in F11 and F13.  I'm not mirroring F12 or Rawhide, but I'm assuming it's the same there.

Jesse, I think this may be related to your fix for finding the old metadata... will have to do some more testing.

Is there anything I can do to help track this down?  Is the problem in mash or another component?  Where should I look and how can I test this?

The simplest way is to run mash in a way that mimics the updates compose process, and see if it happens.

Which version of mash is being used in the compose process?  I'm assuming it's the same for all releases, as they're all hitting the same bug.

0.5.16-1.el5. (0.5.16-1.<otherrelease> should behave the same.)

'K, thanks.  /me goes off to learn how mash works.

Yes, this is a serious bug, happens for Fedora 12 too. In my opinion shud be marked as High priority.

I am a regular user, with slow internet, and rely on delta rpms very much for fast downloading of updates.

Someone should fix this bug immediately, as it increases load on all the mirrors and consumes unnecessary bandwidth.

Just yesterday pidgin delta rpms were there but today they are lost. So something is deleting old delta rpms. Cant they be manually restored?

Thank you,

(In reply to comment #1)
> Just to note, I've noticed this behavior in F11 and F13.  I'm not mirroring F12
> or Rawhide, but I'm assuming it's the same there.    

Yes, its present in F-12 and F-13.

-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

What are the current .mash files being used for F12, F13 and Rawhide?  I've managed to get a local koji repository up and running, and I'm running mash on it, but I don't think I'll be able to reproduce the problem unless I can actually see the config files mash is using the build system.

For rawhide, it uses the config that's in the package.

Ok, but I do also need the .mash files for F12 and/or F13.  With my experimentation, it seems that it may have something to do with the delta_dir option, which isn't used for Rawhide.

# mash config file

[f12-updates]
rpm_path = %(arch)s/
source_path = SRPMS/
debuginfo = True
multilib = True
multilib_method = devel
tag = dist-f12-updates
inherit = False
strict_keys = True
keys = 57BBCCBA
repoviewurl = http://download.fedoraproject.org/pub/fedora/linux/updates/12/%(arch)s/
repoviewtitle = "Fedora 12 Updates - %(arch)s"
arches = i386 x86_64 ppc ppc64
delta = True
delta_dirs = /pub/fedora/linux/releases/12/Everything/%(arch)s/os/,/mnt/koji/mash/updates/f12-updates/%(arch)s/

# mash config file

[f12-updates-testing]
rpm_path = %(arch)s/
source_path = SRPMS/
debuginfo = True
multilib = True
multilib_method = devel
tag = dist-f12-updates-testing
inherit = False
strict_keys = True
keys = 57BBCCBA
repoviewurl = http://download.fedoraproject.org/pub/fedora/linux/updates/testing/12/%(arch)s/
repoviewtitle = "Fedora 12 Updates Testing - %(arch)s"
arches = i386 x86_64 ppc ppc64
delta = True
delta_dirs = /pub/fedora/linux/releases/12/Everything/%(arch)s/os/,/mnt/koji/mash/updates/f12-updates/%(arch)s/

[f13-updates]
rpm_path = %(arch)s/
source_path = SRPMS/
debuginfo = True
multilib = True 
multilib_method = devel
tag = dist-f13-updates
inherit = False
strict_keys = True
keys = E8E40FDE
repoviewurl = http://download.fedoraproject.org/pub/fedora/linux/updates/13/%(arch)s/
repoviewtitle = "Fedora 13 Updates - %(arch)s"
arches = i386 x86_64
delta = True
# Enable this once F13 releases
#delta_dirs = /pub/fedora/linux/releases/13/Everything/%(arch)s/os/,/mnt/koji/mash/updates/f13-updates/%(arch)s/
delta_dirs = /pub/fedora/linux/development/13/%(arch)s/os/

# mash config file

[f13-updates-testing]
rpm_path = %(arch)s/
source_path = SRPMS/
debuginfo = True
multilib = True
multilib_method = devel
tag = dist-f13-updates-testing
inherit = False
strict_keys = True
keys = E8E40FDE
repoviewurl = http://download.fedoraproject.org/pub/fedora/linux/updates/testing/13/%(arch)s/
repoviewtitle = "Fedora 13 Updates Testing - %(arch)s"
arches = i386 x86_64
delta = True
# Enable this once F13 releases
#delta_dirs = /pub/fedora/linux/releases/13/Everything/%(arch)s/os/,/mnt/koji/mash/updates/f13-updates/%(arch)s/
delta_dirs = /pub/fedora/linux/development/13/%(arch)s/os/

Well, there's an obvious bug there in F13.

bodhi config issue filed as https://fedorahosted.org/bodhi/ticket/440

Thanks for filing that.  That explains why deltarpms are only being made against GA (or something close to it) for Fedora 13.

Created attachment 426840 [details]
Save old drpms

Ok, the main problem seems to be that createrepo doesn't keep old deltarpms when it generates a new repository.  To diagram three pushes:

1. foo-1.0.rpm
   drpms/

2. foo-1.1.rpm
   drpms/
       foo-1.0_1.1.drpm

3. foo-1.1.rpm
   bar-1.0.rpm
   drpms/
       
Note that in the third push, foo-1.0_1.1.drpm is missing because it wasn't copied over by createrepo and foo-1.0.rpm is no longer available, so it won't automatically regenerate it.

I've attached a patch against createrepo git master (though it should apply with some fuzz against 0.9.8-2) that will check any drpms in the delta_dirs, and hardlink/copy them if they're still useful.  This also keeps createrepo from regenerating the same deltarpms on each push.

This patch *should* fix the problem, as long as the delta_dirs in the .mash file contain *all* of the rpms and drpms from GA and the last push.

I would assign this to createrepo as that's where the patch needs to be applied, but bugzilla is being strange.  It won't let me change the component.

Could it have something to do with today being F11's EOL?  (The Version should be updated anyway.)

Moving for Jonathan.

And I just mid-aired it.  :)  Thanks much, James.

Just to make sure I understand what you're going for here:


I just tested the behavior you're describing and here's what I'm seeing:

1. nothing in createrepo is pruning old drpms at all. If they are present in the drpms dir they are being added to the presto metadata

2. if I remove the rpms from the old and new pkg dirs the drpms still stick around.

mash does copy old deltas - see _copy_in_deltas in metadata.py.

Copying in of old deltas will be fixed in 0.5.17-1.

mash-0.5.17-1.fc13 has been submitted as an update for Fedora 13.
http://admin.fedoraproject.org/updates/mash-0.5.17-1.fc13

mash-0.5.17-1.el5 has been submitted as an update for Fedora EPEL 5.
http://admin.fedoraproject.org/updates/mash-0.5.17-1.el5

Its been updated only for F13. Does it mean, F12 delta rpms will always get deleted?

No, the key package is the el5 package.  It has been installed on the buildsystem, and it will be used for *all* Fedora releases, so F12 deltarpms will kept from push to push, just like F13 and Rawhide.

ok. thank you for clarification.

Just to clarify - the .el5 version is what's used in bodhi for composing updates. rawhide uses the rawhide mash package.

mash-0.5.17-1.fc13 has been pushed to the Fedora 13 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update mash'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/mash-0.5.17-1.fc13

The main problem here still hasn't been fixed.  All old deltarpms are still being deleted.  For example, R-2.11.0-1.fc13_2.11.1-1.fc13.x86_64.drpm was created for yesterday's push and is no longer available in today's push.  Are the releng machines all running mash-0.5.17?

We are now getting deltarpms against previous updates as well as GA, so at least the bug noticed in comment #14 is fixed now.

No, it's not on the releng boxes at the moment. File an infrastructure ticket, please?

Never mind. Box fixed.

Thank you.

Would it be possible to regenerate all drpms deleted previously?

Not simply. Sorry.

I hate to sound like a broken record, but we've lost 6/29's deltarpms.  Is there more than one releng box that mash can be run on?

Not for updates, no.

This might be asking a lot, but is there any way I could get read-only ssh access to releng?  My local buildsystem isn't showing this bug any more, so I don't think there's much more I can do to track it down without having access to the system where it's happening. :(

Strange.

June 30 drpms exist along with July 1st on this mirror:

ftp://ftp.uci.edu/mirrors/fedora/linux/updates/12/i386/drpms/

But not on this mirror:

http://mirror.cse.iitk.ac.in/fedora/updates/12/i386/drpms/

But June 29 drpms are gone from both.

May be as a temporary fix, mirrors shud not delete files drpms.

(In reply to comment #41)
> Strange.
> 
> June 30 drpms exist along with July 1st on this mirror:
> 
> ftp://ftp.uci.edu/mirrors/fedora/linux/updates/12/i386/drpms/

The drpms have been removed from the official repository, so this mirror either hasn't finished it's sync or isn't using --delete in it's rsync.

> May be as a temporary fix, mirrors should not delete files drpms.    

Unfortunately, if the files aren't in the official repository, then they won't be in prestodelta.xml, which means yum-presto won't use them, even if they are in the drpms directory.

Are the drpms archived/backed up somewhere before deleting?

I would like to use applydeltarpm using those deleted drpms. So I dont have to download huge rpms when I miss 2-3 days for updates.(e.g openoffice)

Thanks in advance.

(In reply to comment #43)
> Are the drpms archived/backed up somewhere before deleting?

No.

Created attachment 432419 [details]
Log old deltarpms that are copied over

Ok, color me stupid for taking this long to realize it, but Rawhide deltarpms are *not* being deleted.  It's only the deltarpms for stable release updates that are being deleted.

Still trying to track down the problem.  Bill, I've attached a patch to mash that will print a list of the deltarpms that we're trying to copy (along with full paths).  Do you mind applying it to mash in releng2?  It only needs to be on for one updates push.

Thanks.

I'm out for a week after today, so I can't commit to that now. If you can wait, I'll see what can be done when I get back; if not, someone else from rel-eng will have to help. Sorry.

I don't know what policies normally need to be followed for a patch like this, so I figured you'd be the best bet for doing it, seeing as you're the mash maintainer.

If this is a fairly routine request, I'll happily ask someone else in rel-eng, but if this is going to generate controversy, I'd rather wait until you get back.

Ok, we've got the patch into releng2's mash, and... absolutely nothing.  None of my logging messages are showing up at all.  I've changed the original patch to use self.logger.info instead of print, and still nothing.

So am I missing something obvious, or is mash for updates *not* being run from the system location?

(In reply to comment #48)
> Ok, we've got the patch into releng2's mash, and... absolutely nothing.  None
> of my logging messages are showing up at all.  I've changed the original patch
> to use self.logger.info instead of print, and still nothing.
> 
> So am I missing something obvious, or is mash for updates *not* being run from
> the system location?    

Bodhi runs the system mash, and writes stdout to mash.out, eg: /mnt/koji/mash/updates/f13-updates-100720.1937/mash.out

Which is what I thought, but color me confused because I've added this line (among others) in /usr/lib/python-2.4/site-packages/mash/metadata.py:

class MetadataNew:
...
    def run(self, path):
        self.conf.outputdir = path
        self.conf.directory = path
        self.repomatic = createrepo.MetaDataGenerator(self.conf)
+       self.logger.info("Previous: %s" % (self.previous,))

but when I run

$ cat /mnt/koji/mash/updates/f13-updates-100720.1937/mash.out | grep Previous
$

there's no output.

Hmm, interesting.  For some reason bodhi only writes stdout from mash after a successful push, yet when it fails it writes stderr as well.  I have seen weird python logging issues in the past that have caused info messages to hit stderr instead of stdout.  So, as a random guess I patched bodhi on releng2 to write out stderr as well, so maybe we'll see these messages after the next run?

New drpms got generated last night under updates/12/i386/drpms/ and all drpm files have modification time of last night. Please check the logs

(In reply to comment #51)
> So, as a random guess I patched bodhi on releng2 to write
> out stderr as well, so maybe we'll see these messages after the next run?    

Thanks for doing that, but I just checked mash.out for the 7/26 run and the extra logging still isn't there.  To be honest, I'm totally lost.

I've finally found it.  While deltarpms do use the original rpm's header as their header, they don't use the original rpm's signatures as their signature.

$ rpm -q --qf "%|DSAHEADER?{%{DSAHEADER:pgpsig}}:HEADER?{%{RSAHEADER:pgpsig}}:{%|SIGGPG?{%{SIGGPG:pgpsig}}:{%|SIGPGP?{%{SIGPGP:pgpsig}}:{(none)}|}|}|}|" -p evolution-2.30.2-1.fc13.x86_64.rpm 
RSA/SHA256, Wed 23 Jun 2010 11:35:37 AM EEST, Key ID 7edc6ad6e8e40fde

$ rpm -q --qf "%|DSAHEADER?{%{DSAHEADER:pgpsig}}:{%|RSAHEADER?{%{RSAHEADER:pgpsig}}:{%|SIGGPG?{%{SIGGPG:pgpsig}}:{%|SIGPGP?{%{SIGPGP:pgpsig}}:{(none)}|}|}|}|" -p evolution-2.30.1-2.fc13_2.30.2-1.fc13.x86_64.drpm 
(none)

However, _sigmatches in metadata.py checks that the signature matches, which it won't do unless the original package wasn't signed (which is why Rawhide was working correctly the last time I checked, but probably isn't now).

A temporary fix would be to disable the signature check in mash on releng2.

I think the best way to permanently fix this is to have python-deltarpm return the target rpm's head as well as everything else.  Unfortunately we're due to lose power here soon (and it's late), so I won't be able to look at this until tomorrow (or possibly the next day).

Crap, this is my fault. This is a known issue that we've carried a patch in the EL-5 builds for, but fell out of the spec file in the rebase.

See bug 512454.

mash-0.5.17-2.fc13 has been pushed to the Fedora 13 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update mash'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/mash-0.5.17-2.fc13

I can confirm that mash-0.5.17-2 fixes the problem.  Bill, could you please push it onto releng2 so that we stop deleting old drpms until I do the promised proper fix?

Working on it.

mash-0.5.20-1.el5 has been submitted as an update for Fedora EPEL 5.
https://admin.fedoraproject.org/updates/mash-0.5.20-1.el5

mash-0.5.20-1.fc13 has been submitted as an update for Fedora 13.
https://admin.fedoraproject.org/updates/mash-0.5.20-1.fc13

mash-0.5.20-1.el5 has been pushed to the Fedora EPEL 5 stable repository.  If problems still persist, please make note of it in this bug report.

mash-0.5.20-1.fc13 has been pushed to the Fedora 13 stable repository.  If problems still persist, please make note of it in this bug report.