Bug 960642

 This is due to the updateinfo on updates/updates-testing being different.

 On updates all the notices are marked as status=stable, but those exact same notices on updates-testing are marked as status=testing.

 So I just put a massive hack into upstream/rawhide/f19 yum to "solve" this on the yum side ... but the problem is actually worse than that, after solving the status problem there are still these left:

An update notice is broken, or duplicate, skipping: updates-testing/FEDORA-2013-1001
An update notice is broken, or duplicate, skipping: updates-testing/FEDORA-2013-2225
An update notice is broken, or duplicate, skipping: updates-testing/FEDORA-2013-0589
An update notice is broken, or duplicate, skipping: updates-testing/FEDORA-2013-0899

...the first 3 of which have different descriptions between updates/updates-testing, but the same ID. The last one has a slightly different "issued" timestamp (in theory I could hack workaround that too ... but I only really want to do that if you can't fix it on your end).

Can we get some movement on this please?  It's a real pain having all of my F19 test machines sending me an email every hour.

It looks like these are all cases where a testing update gets edited and replaced with newer versions, and bodhi doesn't properly prune them from the updateinfo, which it usually does.

I wrote a unit test that reproduces this warning message on F19, as well as a fix to avoid adding duplicate notices to the metadata.

https://github.com/fedora-infra/bodhi/commit/9c3c75f3171086ad2865f7477a716eda7abba490
https://github.com/fedora-infra/bodhi/commit/2815dabbb2fc5f08b08ef5a2dba57b9b02be2e7a

I just wiped out bodhi's repodata cache, which will force it to regenerate the updateinfo from scratch during the next push. This will fix your warnings in the mean time, until I deploy this fix to production.

We got a mail to test@ complaining about one of these this morning, so I guess a new one has showed up.

I just pushed out a new version of bodhi to our releng machines, so hopefully this issue will be resolved after the next push.

Please re-open this bug if you see any more of those broken/duplicate update notice messages from yum.

Getting another now:

An update notice is broken, or duplicate, skipping: updates-testing/FEDORA-2013-10053

Thanks for catching this, Orion.

I made another attempt at avoiding duplicates: https://github.com/fedora-infra/bodhi/commit/8fcda84a24bf386ea005e45f20ca5250facbb2a7

My previous fix seemed to only fix the case where an update is edited and a single build is replaced with a newer version, where as the recent issues arose when an update is edited and new packages are added to it.

Hopefully this'll do the trick. I'll push this fix out to production today.

The latest fix has been in production for a couple of days now. Please reopen this bug if you see those duplicate update notice messages from yum again. Thanks!

New one:

Update notice FEDORA-2013-11685 (from updates-testing) is broken, or a bad duplicate, skipping.
You should report this problem to the owner of the updates-testing repository.

Looks like this is due to an edge case related to the handling of security updates in the update metadata.

I'm working on a fix and will get it deployed after our infrastructure freeze is over. In the mean time, I forced a regeneration of the updateinfo which will happen during the next push.

I don't know if this is the same kind of issue, but am now seeing:

Update notice FEDORA-2013-11662 (from updates-testing) is broken, or a bad

Also:

Update notice FEDORA-2013-12014 (from updates-testing) is broken, or a bad duplicate, skipping.

Another one: 

Update notice FEDORA-2013-12473

Hmm, looks like some of those aren't security related so my theory doesn't hold. I just pushed a patch to git that may solve the problem. If not, it'll have more useful debugging messages.

https://github.com/fedora-infra/bodhi/commit/42adfc63bb5335479a26087589d82e30ee12428d

Ditto to the last three that Orion reported, plus:

FEDORA-2013-12285
FEDORA-2013-12663

(all Update notices from updates-testing)


(In reply to Orion Poplawski from comment #16)
> Another one: 
> 
> Update notice FEDORA-2013-12473

So, in total I now have:


Update notice FEDORA-2013-13468 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-11662 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-12473 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-12663 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-12014 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-7077 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-12285 (from updates-testing) is broken, or a bad duplicate, skipping.

Seem to be down to one now:

Update notice FEDORA-2013-13577 (from updates-testing) is broken, or a bad duplicate, skipping.

New one today:

Update notice FEDORA-2013-14172 (from updates-testing) is broken, or a bad duplicate, skipping.

im also getting these warnings to when i try to look up ( updates-testing ) 

You should report this problem to the owner of the updates-testing repository.
Update notice FEDORA-2013-13577 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-14172 (from updates-testing) is broken, or a bad duplicate, skipping.

Here, I get:
Update notice FEDORA-2013-14172 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-13577 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-14132 (from updates-testing) is broken, or a bad duplicate, skipping.

(Where the last one is new compared to the one reported from above.)

(In reply to Greg` from comment #22)
> im also getting these warnings to when i try to look up ( updates-testing ) 
> 
> You should report this problem to the owner of the updates-testing
> repository.
> Update notice FEDORA-2013-13577 (from updates-testing) is broken, or a bad
> duplicate, skipping.
> Update notice FEDORA-2013-14172 (from updates-testing) is broken, or a bad
> duplicate, skipping.

an update i now get all 3 errors 

Update notice FEDORA-2013-14172 (from updates-testing) is broken, or a bad duplicate, skipping.
You should report this problem to the owner of the updates-testing repository.
Update notice FEDORA-2013-13577 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-14132 (from updates-testing) is broken, or a bad duplicate, skipping.

Same here with the following:

Update notice FEDORA-2013-13577 (from updates-testing) is broken, or a bad duplicate, skipping.
You should report this problem to the owner of the updates-testing repository.
Update notice FEDORA-2013-15734 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-15106 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-14132 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-14172 (from updates-testing) is broken, or a bad duplicate, skipping.

The story continues ---

[frank@localhost ~]$ sudo yum check-update
Loaded plugins: langpacks, refresh-packagekit
Update notice FEDORA-2013-13577 (from updates-testing) is broken, or a bad duplicate, skipping.
You should report this problem to the owner of the updates-testing repository.
Update notice FEDORA-2013-15734 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-15106 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-14132 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-14172 (from updates-testing) is broken, or a bad duplicate, skipping.

So for FEDORA-2013-13577 it looks like it went to stable directly after being edited and a new build added.  The updates-testing debuginfo still has the old version.  It needs to be remove from updates-testing.

 bodhi - 2013-07-24 12:02:04
This update has been submitted for testing by andyp.
bodhi - 2013-07-24 15:40:23
This update is currently being pushed to the Fedora 19 testing updates repository.
bodhi - 2013-07-25 00:46:58
This update has been pushed to testing
bodhi - 2013-07-29 11:25:43
andyp has edited this update. New build(s): gfs2-utils-3.1.6-2.fc19. Removed build(s): gfs2-utils-3.1.6-1.fc19.
bodhi - 2013-07-29 11:25:57
This update has been submitted for testing by andyp.
bodhi - 2013-07-29 15:53:11
This update has reached the stable karma threshold and will be pushed to the stable updates repository
bodhi - 2013-07-29 21:04:16
This update is currently being pushed to the Fedora 19 stable updates repository.
bodhi - 2013-07-30 17:42:42
This update has been pushed to stable

same for me :

[pascal@localhost~]$ sudo yum check-update
[sudo] password for pascal: 
Modules complémentaires chargés : langpacks, refresh-packagekit
adobe-linux-x86_64                                       |  951 B     00:00     
rpmfusion-free-updates                                   | 3.3 kB     00:00     
rpmfusion-free-updates-testing                           | 3.3 kB     00:00     
rpmfusion-nonfree-updates                                | 3.3 kB     00:00     
rpmfusion-nonfree-updates-testing                        | 3.3 kB     00:00     
updates/19/x86_64/metalink                               |  29 kB     00:00     
updates                                                  | 4.6 kB     00:00     
updates-testing/19/x86_64/metalink                       |  27 kB     00:00     
updates-testing                                          | 4.6 kB     00:00     
virtualbox                                               |  951 B     00:00     

Update notice FEDORA-2013-13577(depuis updates-testing) is broken, or a bad duplicate, skipping.
You should report this problem to the owner of the updates-testing repository.
Update notice FEDORA-2013-15734(depuis updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-15106(depuis updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-14132(depuis updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-14172(depuis updates-testing) is broken, or a bad duplicate, skipping.
[pascal@localhost~]$

I just pushed out a new version of bodhi to production, which may fix this issue. If not, I added many more debugging statements to this part of the code which will help me further track this issue down.

Today's crop:

Update notice FEDORA-2013-14132 (from updates-testing) is broken, or a bad duplicate, skipping.
You should report this problem to the owner of the updates-testing repository.
Update notice FEDORA-2013-17026 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-13577 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-15106 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-15734 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-17127 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-14172 (from updates-testing) is broken, or a bad duplicate, skipping.

The latest ( Sat Sept 21-2013)

Update notice FEDORA-2013-14132 (from updates-testing) is broken, or a bad duplicate, skipping.
You should report this problem to the owner of the updates-testing repository.
Update notice FEDORA-2013-17026 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-13577 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-15106 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-15734 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-17127 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-14172 (from updates-testing) is broken, or a bad duplicate, skipping.

Todays (Sept 23rd) batch


[root@localhost frank]# yum check-update
Loaded plugins: langpacks, refresh-packagekit
Update notice FEDORA-2013-15106 (from updates-testing) is broken, or a bad duplicate, skipping.
You should report this problem to the owner of the updates-testing repository.
Update notice FEDORA-2013-17373 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-14172 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-17026 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-17127 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-14132 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-13577 (from updates-testing) is broken, or a bad duplicate, skipping.
Update notice FEDORA-2013-15734 (from updates-testing) is broken, or a bad duplicate, skipping.
[root@localhost frank]# yum update

I wrote a little script that pulls in the updateinfo.xml from updates-testing, throws it at `yum.update_md.UpdateMetadata`, and checks it for duplicate IDs.

    https://gist.github.com/lmacken/6676152

Yum spit out the duplicate/broken error message, but it doesn't actually
contain any notices with duplicate update IDs.

This recent commit to yum added additional logic to compare two
notices using the new UpdateNotice.__eq__ method. 

    http://yum.baseurl.org/gitweb?p=yum.git;a=blob;f=yum/update_md.py;h=cf0b8827d9c45af76d5ac7070b29799c5801b1cc;hb=HEAD#l483

Looking closer at this new comparison method I discovered that "identical" update notices were actually failing this check because the 'issued' timestamps were different: '2013-09-22 03:08:42' != '2013-09-22 03:08:42.977048'.

I just committed this change to bodhi to properly format the timestamp, as
opposed to just using datetime.__unicode__. Hopefully that will prevent any extra precision from sneaking in.

    https://github.com/fedora-infra/bodhi/commit/43f4b8223fb2aee671029696e80d1af942fef12d

Infrastructure unfreezes on Wednesday, so I'll try and get this pushed out by then.

The yum commit I was referencing: http://yum.baseurl.org/gitweb?p=yum.git;a=patch;h=2ebc8ab8e9043985acb9da530af8269e1aafc446

I noticed another warning after today's push, which is due to the security update edge case that I mentioned in Comment #13. I'm working on a fix for this now.

I haven't had any of these now for a few days. Hopefully fixed.

I hopefully fixed the last edge-case for this issue in https://github.com/fedora-infra/bodhi/commit/618d5de42c8377b4fbd56aa94122a20c6a3bc76e

Please re-open this bug if you see the yum warning again. Thanks!

New one:

Update notice FEDORA-2013-18382 (from updates-testing) is broken, or a bad duplicate, skipping.

Thanks Orion, I think I see what is happening.

I just pushed a few changes to git that will hopefully take care of the problem. I'll deploy it as soon as the current push finishes.

http://git.io/OxeLeQ

Yum check-update today Sunday Oct 13

Update notice FEDORA-2013-18382 (from updates-testing) is broken, or a bad duplicate, skipping.
You should report this problem to the owner of the updates-testing repository.

My latest fixes went out to production yesterday. Please re-open this bug if you notice this warning from yum again.

*** Bug 1008067 has been marked as a duplicate of this bug. ***

> resolved as CURRENTRELEASE 

how comes?

[root@srv-rhsoft:~]$ date; yum check-update; date 
Mi 30. Okt 00:59:35 CET 2013
Geladene Plugins: etckeeper, protectbase, tsflags  
0 packages excluded due to repository protections
Update notice FEDORA-2013-19479 (von updates-testing) is broken, or a bad duplicate, skipping.
You should report this problem to the owner of the updates-testing repository.
Mi 30. Okt 00:59:44 CET 2013

-------- Original-Nachricht --------
Betreff: Re: Update notice FEDORA-2013-13577 (von updates-testing) is broken, 	or a bad duplicate, skipping
Datum: Wed, 30 Oct 2013 00:49:36 +0100
Von: Reindl Harald <h.reindl>
An: devel.org

Am 14.09.2013 03:37, schrieb Theodore Lee:
> On 14/09/13 08:51, Reindl Harald wrote:
>> a few days ago i upgraded to F19 and my cron-script checking
>> for updates as well as "yum check-update" reports these
>> warnings - unsure where to report a bug because it's not
>> a specific package, so i post it here
>>
>> Update notice FEDORA-2013-13577 (von updates-testing) is broken, or a bad duplicate, skipping.
>> You should report this problem to the owner of the updates-testing repository.
>> Update notice FEDORA-2013-15734 (von updates-testing) is broken, or a bad duplicate, skipping.
>> Update notice FEDORA-2013-15106 (von updates-testing) is broken, or a bad duplicate, skipping.
>> Update notice FEDORA-2013-14132 (von updates-testing) is broken, or a bad duplicate, skipping.
>> Update notice FEDORA-2013-14172 (von updates-testing) is broken, or a bad duplicate, skipping.
> 
> Evidently this is a bodhi bug that's been around for a while, being
> tracked in bug 960642.[1]

and will this ever be fixed?

-------- Original-Nachricht --------
Betreff: Anacron job 'cron.daily' on srv-rhsoft.rhsoft.net
Datum: Wed, 30 Oct 2013 00:08:53 +0100 (CET)
Von: Anacron <root.net>
An: root.net

/etc/cron.daily/check-updates.cron:

Update notice FEDORA-2013-19479 (from-testing) is broken, or a bad duplicate, skipping. You should report this
problem to the owner of the-testing repository.

reopening to investigate...

I'm getting this:

$ sudo yum --enablerepo=updates-testing check-update
...
Update notice FEDORA-2013-19479 (from updates-testing) is broken, or a bad duplicate, skipping.
You should report this problem to the owner of the updates-testing repository.

This latest one is due to a couple of bugs working together. First, bodhi was not properly purging updates from the testing metadata once they hit stable. It also wasn't regenerating notices for updates that have been edited. So with 2 update notices with the same ID in 2 different repos with different descriptions, yum gets confused and spits out that warning.

Today I refactored and simplified the updateinfo caching code and added more unit tests. We're frozen until next week, but I'll see what I can do about getting approval to deploy it before then, or at least flush bodhi's repodata cache to clear up the current ones.

https://github.com/fedora-infra/bodhi/compare/70d990e88436...221b57220564

They're back !!

[root@localhost frank]# yum check-update
Loaded plugins: langpacks, refresh-packagekit
Update notice FEDORA-2013-20814 (from updates-testing) is broken, or a bad duplicate, skipping.
You should report this problem to the owner of the updates-testing repository.
Update notice FEDORA-2013-20736 (from updates-testing) is broken, or a bad duplicate, skipping.
[root@localhost frank]# 


First time today in a longtime.

I just pushed a new version of bodhi out that will hopefully fix this issue. Please re-open this bug if you notice this warning message again, thanks!

it is happening the same with RedHat version:
cat /etc/redhat-release 
Red Hat Enterprise Linux Workstation release 6.4 (Santiago)


Command: yum update -y

Output:
An update notice is broken, or duplicate, skipping: RHEA-2013:1774
Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in <module>
    yummain.user_main(sys.argv[1:], exit_code=True)
  File "/usr/share/yum-cli/yummain.py", line 285, in user_main
    errcode = main(args)
  File "/usr/share/yum-cli/yummain.py", line 219, in main
    return_code = base.doTransaction()
  File "/usr/share/yum-cli/cli.py", line 498, in doTransaction
    problems = self.downloadPkgs(downloadpkgs, callback_total=self.download_callback_total_cb) 
  File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 2088, in downloadPkgs
    self.plugins.run('postdownload', pkglist=pkglist, errors=errors)
  File "/usr/lib/python2.6/site-packages/yum/plugins.py", line 184, in run
    func(conduitcls(self, self.base, conf, **kwargs))
  File "/usr/lib/yum-plugins/ibm-check-lotus-updates.py", line 259, in postdownload_hook
    ISLOTUSPKGSEC = isLotusUpdateSecurity( conduit, pkgtups )
  File "/usr/lib/yum-plugins/ibm-check-lotus-updates.py", line 83, in isLotusUpdateSecurity
    md_info.add(repo)
  File "/usr/lib/python2.6/site-packages/yum/update_md.py", line 528, in add
    if not self.add_notice(un):
  File "/usr/lib/python2.6/site-packages/yum/update_md.py", line 479, in add_notice
    seen.add(pkg['name'])
KeyError: 'name'

I can confirm that I am seeing the same thing as in Comment 49 on a RHEL 6.4 box - same error for release RHEA-2013:1774 when I try a full upgrade and some single package upgrades.  and the exact same error message.

The problem from Comment #49 seems to be something else (furthermore there seems to be some ibm plugin installed). I guess it would be best to open a separate bug for that...

Today it seems that kernel-3.12.8-300.fc20 was moved from updates-testing to updates, but wasn't deleted (yet?) from updates-testing and this warning appears:

# yum --enablerepo updates-testing check-update 
[snip]
Update notice FEDORA-2014-1062 (from updates-testing) is broken, or a bad duplicate, skipping.
You should report this problem to the owner of the updates-testing repository.

A "yum clean metadata" doesn't seem to help.

(In reply to Thomas Spura from comment #51)
> The problem from Comment #49 seems to be something else (furthermore there
> seems to be some ibm plugin installed). I guess it would be best to open a
> separate bug for that...

Yeah, that exception looks odd. I'm wondering if there is an IBM repo with some malformed updateinfo.xml in the repodata.

> Today it seems that kernel-3.12.8-300.fc20 was moved from updates-testing to
> updates, but wasn't deleted (yet?) from updates-testing and this warning
> appears:
> 
> # yum --enablerepo updates-testing check-update 
> [snip]
> Update notice FEDORA-2014-1062 (from updates-testing) is broken, or a bad
> duplicate, skipping.
> You should report this problem to the owner of the updates-testing
> repository.
> 
> A "yum clean metadata" doesn't seem to help.

I cannot reproduce this issue, are you still able to? I'm only seeing that kernel update in the stable metadata, not in the testing. Maybe this is a race condition between syncing the stable/testing repos to the mirrors?

I'm using this script to try and reproduce the issue: https://gist.github.com/lmacken/6676152

(In reply to martingfernandez from comment #49)
> it is happening the same with RedHat version:
> cat /etc/redhat-release 
> Red Hat Enterprise Linux Workstation release 6.4 (Santiago)
> 
> 
> Command: yum update -y
> 
> Output:
> An update notice is broken, or duplicate, skipping: RHEA-2013:1774
> Traceback (most recent call last):
>   File "/usr/bin/yum", line 29, in <module>
>     yummain.user_main(sys.argv[1:], exit_code=True)
>   File "/usr/share/yum-cli/yummain.py", line 285, in user_main
>     errcode = main(args)
>   File "/usr/share/yum-cli/yummain.py", line 219, in main
>     return_code = base.doTransaction()
>   File "/usr/share/yum-cli/cli.py", line 498, in doTransaction
>     problems = self.downloadPkgs(downloadpkgs,
> callback_total=self.download_callback_total_cb) 
>   File "/usr/lib/python2.6/site-packages/yum/__init__.py", line 2088, in
> downloadPkgs
>     self.plugins.run('postdownload', pkglist=pkglist, errors=errors)
>   File "/usr/lib/python2.6/site-packages/yum/plugins.py", line 184, in run
>     func(conduitcls(self, self.base, conf, **kwargs))
>   File "/usr/lib/yum-plugins/ibm-check-lotus-updates.py", line 259, in
> postdownload_hook
>     ISLOTUSPKGSEC = isLotusUpdateSecurity( conduit, pkgtups )
>   File "/usr/lib/yum-plugins/ibm-check-lotus-updates.py", line 83, in
> isLotusUpdateSecurity
>     md_info.add(repo)
>   File "/usr/lib/python2.6/site-packages/yum/update_md.py", line 528, in add
>     if not self.add_notice(un):
>   File "/usr/lib/python2.6/site-packages/yum/update_md.py", line 479, in
> add_notice
>     seen.add(pkg['name'])
> KeyError: 'name'


This is a different issue. Could you please open a new bug against RHEL for this? Thanks

(In reply to Luke Macken from comment #52)
> (In reply to Thomas Spura from comment #51)
> > # yum --enablerepo updates-testing check-update 
> > [snip]
> > Update notice FEDORA-2014-1062 (from updates-testing) is broken, or a bad
> > duplicate, skipping.
> > You should report this problem to the owner of the updates-testing
> > repository.
> > 
> > A "yum clean metadata" doesn't seem to help.
> 
> I cannot reproduce this issue, are you still able to? I'm only seeing that
> kernel update in the stable metadata, not in the testing. Maybe this is a
> race condition between syncing the stable/testing repos to the mirrors?

It seems to. IIRC, stable/testing repos came from two different mirrors, so this is likely a race condition between the two. I don't think that can be fixed by yum or so...

Thanks!