Bug 598584

Summary: Deltarpms are deleted from repository while target rpms are still in it
Product: [Fedora] Fedora Reporter: Jonathan Dieter <jdieter>
Component: mashAssignee: Bill Nottingham <notting>
Status: CLOSED ERRATA QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: low    
Version: 13CC: anon.amish, benjavalero, chris.brown, dcantrell, gspurgeon, james.antill, jwboyer, lmacken, mohd.izhar.firdaus, nareshsukhija, notting, robatino, rvokal, yehielb
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: mash-0.5.20-1.fc13 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2010-10-13 05:59:15 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
Attachments:
Description Flags
Save old drpms
none
Log old deltarpms that are copied over none

Description Jonathan Dieter 2010-06-01 16:46:11 UTC
Deltarpms are being deleted from repositories while their target rpms are still in the repository.  For example, there were F13 GA -> updates-released openoffice.org deltarpms that have now been deleted, even though the rpms are still in updates.

It seems that after every push, all of the old deltarpms are deleted, and the only deltarpms left are those for whatever packages were in the current push.  I'm assigning this to mash, please move elsewhere if it's the wrong component.

Comment 1 Jonathan Dieter 2010-06-01 16:47:28 UTC
Just to note, I've noticed this behavior in F11 and F13.  I'm not mirroring F12 or Rawhide, but I'm assuming it's the same there.

Comment 2 Bill Nottingham 2010-06-01 21:49:06 UTC
Jesse, I think this may be related to your fix for finding the old metadata... will have to do some more testing.

Comment 3 Jonathan Dieter 2010-06-09 06:40:11 UTC
Is there anything I can do to help track this down?  Is the problem in mash or another component?  Where should I look and how can I test this?

Comment 4 Bill Nottingham 2010-06-09 17:42:35 UTC
The simplest way is to run mash in a way that mimics the updates compose process, and see if it happens.

Comment 5 Jonathan Dieter 2010-06-09 17:51:39 UTC
Which version of mash is being used in the compose process?  I'm assuming it's the same for all releases, as they're all hitting the same bug.

Comment 6 Bill Nottingham 2010-06-09 17:56:20 UTC
0.5.16-1.el5. (0.5.16-1.<otherrelease> should behave the same.)

Comment 7 Jonathan Dieter 2010-06-09 18:02:35 UTC
'K, thanks.  /me goes off to learn how mash works.

Comment 8 AMM 2010-06-12 02:05:16 UTC
Yes, this is a serious bug, happens for Fedora 12 too. In my opinion shud be marked as High priority.

I am a regular user, with slow internet, and rely on delta rpms very much for fast downloading of updates.

Someone should fix this bug immediately, as it increases load on all the mirrors and consumes unnecessary bandwidth.

Just yesterday pidgin delta rpms were there but today they are lost. So something is deleting old delta rpms. Cant they be manually restored?

Thank you,

Comment 9 Christopher Brown 2010-06-13 16:32:02 UTC
(In reply to comment #1)
> Just to note, I've noticed this behavior in F11 and F13.  I'm not mirroring F12
> or Rawhide, but I'm assuming it's the same there.    

Yes, its present in F-12 and F-13.

Comment 10 Benjamín Valero Espinosa 2010-06-15 08:14:06 UTC

-- 
Fedora Bugzappers volunteer triage team
https://fedoraproject.org/wiki/BugZappers

Comment 11 Jonathan Dieter 2010-06-19 06:27:05 UTC
What are the current .mash files being used for F12, F13 and Rawhide?  I've managed to get a local koji repository up and running, and I'm running mash on it, but I don't think I'll be able to reproduce the problem unless I can actually see the config files mash is using the build system.

Comment 12 Bill Nottingham 2010-06-21 19:10:56 UTC
For rawhide, it uses the config that's in the package.

Comment 13 Jonathan Dieter 2010-06-22 19:18:24 UTC
Ok, but I do also need the .mash files for F12 and/or F13.  With my experimentation, it seems that it may have something to do with the delta_dir option, which isn't used for Rawhide.

Comment 14 Bill Nottingham 2010-06-22 19:23:31 UTC
# mash config file

[f12-updates]
rpm_path = %(arch)s/
source_path = SRPMS/
debuginfo = True
multilib = True
multilib_method = devel
tag = dist-f12-updates
inherit = False
strict_keys = True
keys = 57BBCCBA
repoviewurl = http://download.fedoraproject.org/pub/fedora/linux/updates/12/%(arch)s/
repoviewtitle = "Fedora 12 Updates - %(arch)s"
arches = i386 x86_64 ppc ppc64
delta = True
delta_dirs = /pub/fedora/linux/releases/12/Everything/%(arch)s/os/,/mnt/koji/mash/updates/f12-updates/%(arch)s/

# mash config file

[f12-updates-testing]
rpm_path = %(arch)s/
source_path = SRPMS/
debuginfo = True
multilib = True
multilib_method = devel
tag = dist-f12-updates-testing
inherit = False
strict_keys = True
keys = 57BBCCBA
repoviewurl = http://download.fedoraproject.org/pub/fedora/linux/updates/testing/12/%(arch)s/
repoviewtitle = "Fedora 12 Updates Testing - %(arch)s"
arches = i386 x86_64 ppc ppc64
delta = True
delta_dirs = /pub/fedora/linux/releases/12/Everything/%(arch)s/os/,/mnt/koji/mash/updates/f12-updates/%(arch)s/

[f13-updates]
rpm_path = %(arch)s/
source_path = SRPMS/
debuginfo = True
multilib = True 
multilib_method = devel
tag = dist-f13-updates
inherit = False
strict_keys = True
keys = E8E40FDE
repoviewurl = http://download.fedoraproject.org/pub/fedora/linux/updates/13/%(arch)s/
repoviewtitle = "Fedora 13 Updates - %(arch)s"
arches = i386 x86_64
delta = True
# Enable this once F13 releases
#delta_dirs = /pub/fedora/linux/releases/13/Everything/%(arch)s/os/,/mnt/koji/mash/updates/f13-updates/%(arch)s/
delta_dirs = /pub/fedora/linux/development/13/%(arch)s/os/

# mash config file

[f13-updates-testing]
rpm_path = %(arch)s/
source_path = SRPMS/
debuginfo = True
multilib = True
multilib_method = devel
tag = dist-f13-updates-testing
inherit = False
strict_keys = True
keys = E8E40FDE
repoviewurl = http://download.fedoraproject.org/pub/fedora/linux/updates/testing/13/%(arch)s/
repoviewtitle = "Fedora 13 Updates Testing - %(arch)s"
arches = i386 x86_64
delta = True
# Enable this once F13 releases
#delta_dirs = /pub/fedora/linux/releases/13/Everything/%(arch)s/os/,/mnt/koji/mash/updates/f13-updates/%(arch)s/
delta_dirs = /pub/fedora/linux/development/13/%(arch)s/os/

Well, there's an obvious bug there in F13.

Comment 15 Bill Nottingham 2010-06-22 19:25:31 UTC
bodhi config issue filed as https://fedorahosted.org/bodhi/ticket/440

Comment 16 Jonathan Dieter 2010-06-22 19:38:50 UTC
Thanks for filing that.  That explains why deltarpms are only being made against GA (or something close to it) for Fedora 13.

Comment 17 Jonathan Dieter 2010-06-25 10:27:01 UTC
Created attachment 426840 [details]
Save old drpms

Ok, the main problem seems to be that createrepo doesn't keep old deltarpms when it generates a new repository.  To diagram three pushes:

1. foo-1.0.rpm
   drpms/

2. foo-1.1.rpm
   drpms/
       foo-1.0_1.1.drpm

3. foo-1.1.rpm
   bar-1.0.rpm
   drpms/
       
Note that in the third push, foo-1.0_1.1.drpm is missing because it wasn't copied over by createrepo and foo-1.0.rpm is no longer available, so it won't automatically regenerate it.

I've attached a patch against createrepo git master (though it should apply with some fuzz against 0.9.8-2) that will check any drpms in the delta_dirs, and hardlink/copy them if they're still useful.  This also keeps createrepo from regenerating the same deltarpms on each push.

This patch *should* fix the problem, as long as the delta_dirs in the .mash file contain *all* of the rpms and drpms from GA and the last push.

Comment 18 Jonathan Dieter 2010-06-25 10:30:53 UTC
I would assign this to createrepo as that's where the patch needs to be applied, but bugzilla is being strange.  It won't let me change the component.

Comment 19 Andre Robatino 2010-06-25 10:37:08 UTC
Could it have something to do with today being F11's EOL?  (The Version should be updated anyway.)

Comment 20 James Antill 2010-06-25 13:26:05 UTC
Moving for Jonathan.

Comment 21 Jonathan Dieter 2010-06-25 13:29:20 UTC
And I just mid-aired it.  :)  Thanks much, James.

Comment 22 seth vidal 2010-06-25 14:23:42 UTC
Just to make sure I understand what you're going for here:


I just tested the behavior you're describing and here's what I'm seeing:

1. nothing in createrepo is pruning old drpms at all. If they are present in the drpms dir they are being added to the presto metadata

2. if I remove the rpms from the old and new pkg dirs the drpms still stick around.

Comment 23 Bill Nottingham 2010-06-25 15:25:31 UTC
mash does copy old deltas - see _copy_in_deltas in metadata.py.

Comment 24 Bill Nottingham 2010-06-25 16:42:48 UTC
Copying in of old deltas will be fixed in 0.5.17-1.

Comment 25 Fedora Update System 2010-06-25 16:46:53 UTC
mash-0.5.17-1.fc13 has been submitted as an update for Fedora 13.
http://admin.fedoraproject.org/updates/mash-0.5.17-1.fc13

Comment 26 Fedora Update System 2010-06-25 16:49:28 UTC
mash-0.5.17-1.el5 has been submitted as an update for Fedora EPEL 5.
http://admin.fedoraproject.org/updates/mash-0.5.17-1.el5

Comment 27 AMM 2010-06-26 04:36:16 UTC
Its been updated only for F13. Does it mean, F12 delta rpms will always get deleted?

Comment 28 Jonathan Dieter 2010-06-26 05:03:38 UTC
No, the key package is the el5 package.  It has been installed on the buildsystem, and it will be used for *all* Fedora releases, so F12 deltarpms will kept from push to push, just like F13 and Rawhide.

Comment 29 AMM 2010-06-26 07:26:19 UTC
ok. thank you for clarification.

Comment 30 Bill Nottingham 2010-06-28 17:14:07 UTC
Just to clarify - the .el5 version is what's used in bodhi for composing updates. rawhide uses the rawhide mash package.

Comment 31 Fedora Update System 2010-06-28 17:19:53 UTC
mash-0.5.17-1.fc13 has been pushed to the Fedora 13 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update mash'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/mash-0.5.17-1.fc13

Comment 32 Jonathan Dieter 2010-06-29 20:52:41 UTC
The main problem here still hasn't been fixed.  All old deltarpms are still being deleted.  For example, R-2.11.0-1.fc13_2.11.1-1.fc13.x86_64.drpm was created for yesterday's push and is no longer available in today's push.  Are the releng machines all running mash-0.5.17?

We are now getting deltarpms against previous updates as well as GA, so at least the bug noticed in comment #14 is fixed now.

Comment 33 Bill Nottingham 2010-06-29 21:19:48 UTC
No, it's not on the releng boxes at the moment. File an infrastructure ticket, please?

Comment 34 Bill Nottingham 2010-06-29 21:24:50 UTC
Never mind. Box fixed.

Comment 35 Jonathan Dieter 2010-06-29 21:26:40 UTC
Thank you.

Comment 36 AMM 2010-06-30 01:27:59 UTC
Would it be possible to regenerate all drpms deleted previously?

Comment 37 Bill Nottingham 2010-06-30 16:27:37 UTC
Not simply. Sorry.

Comment 38 Jonathan Dieter 2010-06-30 19:41:04 UTC
I hate to sound like a broken record, but we've lost 6/29's deltarpms.  Is there more than one releng box that mash can be run on?

Comment 39 Bill Nottingham 2010-06-30 19:59:39 UTC
Not for updates, no.

Comment 40 Jonathan Dieter 2010-06-30 20:35:32 UTC
This might be asking a lot, but is there any way I could get read-only ssh access to releng?  My local buildsystem isn't showing this bug any more, so I don't think there's much more I can do to track it down without having access to the system where it's happening. :(

Comment 41 AMM 2010-07-02 04:11:02 UTC
Strange.

June 30 drpms exist along with July 1st on this mirror:

ftp://ftp.uci.edu/mirrors/fedora/linux/updates/12/i386/drpms/

But not on this mirror:

http://mirror.cse.iitk.ac.in/fedora/updates/12/i386/drpms/

But June 29 drpms are gone from both.

May be as a temporary fix, mirrors shud not delete files drpms.

Comment 42 Jonathan Dieter 2010-07-02 11:00:46 UTC
(In reply to comment #41)
> Strange.
> 
> June 30 drpms exist along with July 1st on this mirror:
> 
> ftp://ftp.uci.edu/mirrors/fedora/linux/updates/12/i386/drpms/

The drpms have been removed from the official repository, so this mirror either hasn't finished it's sync or isn't using --delete in it's rsync.

> May be as a temporary fix, mirrors should not delete files drpms.    

Unfortunately, if the files aren't in the official repository, then they won't be in prestodelta.xml, which means yum-presto won't use them, even if they are in the drpms directory.

Comment 43 AMM 2010-07-09 03:39:51 UTC
Are the drpms archived/backed up somewhere before deleting?

I would like to use applydeltarpm using those deleted drpms. So I dont have to download huge rpms when I miss 2-3 days for updates.(e.g openoffice)

Thanks in advance.

Comment 44 Bill Nottingham 2010-07-09 17:38:49 UTC
(In reply to comment #43)
> Are the drpms archived/backed up somewhere before deleting?

No.

Comment 45 Jonathan Dieter 2010-07-16 15:44:40 UTC
Created attachment 432419 [details]
Log old deltarpms that are copied over

Ok, color me stupid for taking this long to realize it, but Rawhide deltarpms are *not* being deleted.  It's only the deltarpms for stable release updates that are being deleted.

Still trying to track down the problem.  Bill, I've attached a patch to mash that will print a list of the deltarpms that we're trying to copy (along with full paths).  Do you mind applying it to mash in releng2?  It only needs to be on for one updates push.

Thanks.

Comment 46 Bill Nottingham 2010-07-16 15:58:16 UTC
I'm out for a week after today, so I can't commit to that now. If you can wait, I'll see what can be done when I get back; if not, someone else from rel-eng will have to help. Sorry.

Comment 47 Jonathan Dieter 2010-07-16 16:11:35 UTC
I don't know what policies normally need to be followed for a patch like this, so I figured you'd be the best bet for doing it, seeing as you're the mash maintainer.

If this is a fairly routine request, I'll happily ask someone else in rel-eng, but if this is going to generate controversy, I'd rather wait until you get back.

Comment 48 Jonathan Dieter 2010-07-22 18:28:21 UTC
Ok, we've got the patch into releng2's mash, and... absolutely nothing.  None of my logging messages are showing up at all.  I've changed the original patch to use self.logger.info instead of print, and still nothing.

So am I missing something obvious, or is mash for updates *not* being run from the system location?

Comment 49 Luke Macken 2010-07-22 21:31:27 UTC
(In reply to comment #48)
> Ok, we've got the patch into releng2's mash, and... absolutely nothing.  None
> of my logging messages are showing up at all.  I've changed the original patch
> to use self.logger.info instead of print, and still nothing.
> 
> So am I missing something obvious, or is mash for updates *not* being run from
> the system location?    

Bodhi runs the system mash, and writes stdout to mash.out, eg: /mnt/koji/mash/updates/f13-updates-100720.1937/mash.out

Comment 50 Jonathan Dieter 2010-07-23 04:44:34 UTC
Which is what I thought, but color me confused because I've added this line (among others) in /usr/lib/python-2.4/site-packages/mash/metadata.py:

class MetadataNew:
...
    def run(self, path):
        self.conf.outputdir = path
        self.conf.directory = path
        self.repomatic = createrepo.MetaDataGenerator(self.conf)
+       self.logger.info("Previous: %s" % (self.previous,))

but when I run

$ cat /mnt/koji/mash/updates/f13-updates-100720.1937/mash.out | grep Previous
$

there's no output.

Comment 51 Luke Macken 2010-07-23 05:47:17 UTC
Hmm, interesting.  For some reason bodhi only writes stdout from mash after a successful push, yet when it fails it writes stderr as well.  I have seen weird python logging issues in the past that have caused info messages to hit stderr instead of stdout.  So, as a random guess I patched bodhi on releng2 to write out stderr as well, so maybe we'll see these messages after the next run?

Comment 52 Naresh Sukhija 2010-07-27 12:11:07 UTC
New drpms got generated last night under updates/12/i386/drpms/ and all drpm files have modification time of last night. Please check the logs

Comment 53 Jonathan Dieter 2010-07-27 17:24:05 UTC
(In reply to comment #51)
> So, as a random guess I patched bodhi on releng2 to write
> out stderr as well, so maybe we'll see these messages after the next run?    

Thanks for doing that, but I just checked mash.out for the 7/26 run and the extra logging still isn't there.  To be honest, I'm totally lost.

Comment 54 Jonathan Dieter 2010-07-28 19:22:50 UTC
I've finally found it.  While deltarpms do use the original rpm's header as their header, they don't use the original rpm's signatures as their signature.

$ rpm -q --qf "%|DSAHEADER?{%{DSAHEADER:pgpsig}}:HEADER?{%{RSAHEADER:pgpsig}}:{%|SIGGPG?{%{SIGGPG:pgpsig}}:{%|SIGPGP?{%{SIGPGP:pgpsig}}:{(none)}|}|}|}|" -p evolution-2.30.2-1.fc13.x86_64.rpm 
RSA/SHA256, Wed 23 Jun 2010 11:35:37 AM EEST, Key ID 7edc6ad6e8e40fde

$ rpm -q --qf "%|DSAHEADER?{%{DSAHEADER:pgpsig}}:{%|RSAHEADER?{%{RSAHEADER:pgpsig}}:{%|SIGGPG?{%{SIGGPG:pgpsig}}:{%|SIGPGP?{%{SIGPGP:pgpsig}}:{(none)}|}|}|}|" -p evolution-2.30.1-2.fc13_2.30.2-1.fc13.x86_64.drpm 
(none)

However, _sigmatches in metadata.py checks that the signature matches, which it won't do unless the original package wasn't signed (which is why Rawhide was working correctly the last time I checked, but probably isn't now).

A temporary fix would be to disable the signature check in mash on releng2.

I think the best way to permanently fix this is to have python-deltarpm return the target rpm's head as well as everything else.  Unfortunately we're due to lose power here soon (and it's late), so I won't be able to look at this until tomorrow (or possibly the next day).

Comment 55 Bill Nottingham 2010-07-28 19:47:13 UTC
Crap, this is my fault. This is a known issue that we've carried a patch in the EL-5 builds for, but fell out of the spec file in the rebase.

Comment 56 Bill Nottingham 2010-07-28 19:47:56 UTC
See bug 512454.

Comment 57 Fedora Update System 2010-08-03 00:35:30 UTC
mash-0.5.17-2.fc13 has been pushed to the Fedora 13 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update mash'.  You can provide feedback for this update here: http://admin.fedoraproject.org/updates/mash-0.5.17-2.fc13

Comment 58 Jonathan Dieter 2010-08-07 22:54:42 UTC
I can confirm that mash-0.5.17-2 fixes the problem.  Bill, could you please push it onto releng2 so that we stop deleting old drpms until I do the promised proper fix?

Comment 59 Bill Nottingham 2010-08-09 17:22:54 UTC
Working on it.

Comment 60 Fedora Update System 2010-09-28 16:19:08 UTC
mash-0.5.20-1.el5 has been submitted as an update for Fedora EPEL 5.
https://admin.fedoraproject.org/updates/mash-0.5.20-1.el5

Comment 61 Fedora Update System 2010-09-28 16:33:47 UTC
mash-0.5.20-1.fc13 has been submitted as an update for Fedora 13.
https://admin.fedoraproject.org/updates/mash-0.5.20-1.fc13

Comment 62 Fedora Update System 2010-10-13 05:57:43 UTC
mash-0.5.20-1.el5 has been pushed to the Fedora EPEL 5 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 63 Fedora Update System 2011-04-14 20:58:22 UTC
mash-0.5.20-1.fc13 has been pushed to the Fedora 13 stable repository.  If problems still persist, please make note of it in this bug report.