Bug 1124533

Thank you for filing this! I had noticed this but had not spent the time to investigate.

The root cause seems to be that fedmsg is publishing multiple messages for each build. For instance, I see the following three messages for systemd-208-20.fc20:

  timestamp        build_id  task_id  instance  owner
  ---------        --------  -------  --------  -----
  2014-07-20T00:23 545909    7167965  primary   zbyszek
  2014-07-21T01:16 254412    1441395  s390      sharkcz
  2014-07-21T04:14 252906    1957605  ppc       karsten

(Above table shows only diffs. Common fields were N-V-R, attribute="state", old="0", new="1").

I've been unable to find documentation on the message structure but my gut feel is that "instance=primary" is the key factor here. I've updated the listener script so it will only queue jobs where instance=primary.

This will only affect runs as of this instance in time: I'm afraid past runs will remain as they are, with multiple runs for an NVR and bad build IDs in those runs.

https://git.fedorahosted.org/cgit/rpmgrill.git/commit/?h=scaffolding&id=e1b41521b9a336ccea3467fd779d3666fbf9658b

Thanks for quick response.

The additional builds are on different koji instances. I guess that's why the build numbers don't match.

http://s390.koji.fedoraproject.org/koji/buildinfo?buildID=254412 is indeed  systemd-208-20.fc20, and
http://ppc.koji.fedoraproject.org/koji/buildinfo?buildID=252906 matches too.

Oh, yikes. This certainly makes things interesting! I had assumed that a koji build notification meant "all builds on all architectures are done".

rpmgrill is designed to analyze all builds at once, meaning all possible architectures; it assumes that "koji download-build" fetches all built RPMs. This concept of instances breaks my assumptions. For now, we'll have to live with x86- and noarch-only analysis. The Taskotron effort should take this into consideration.

Thanks for your followup, it greatly clarified my understanding of the koji environment.

(In reply to Ed Santiago from comment #3)
> Oh, yikes. This certainly makes things interesting! I had assumed that a
> koji build notification meant "all builds on all architectures are done".
AFAIU, build on those secondary architectures (or whatever the correct term is) are independent. I think that they are scheduled by some bot, possibly with a substantial delay. So I don't think that handling them together is possible.

E.g., I see the following notification:

00:10 <fedora-notifs> buildsys.build.state.change -- sharkcz's systemd-213-1.fc21 started building (s390) http://s390.koji.fedoraproject.org/koji/buildinfo?buildID=254655

But systemd-215-4.fc21 was built recently on the main architectures.

> rpmgrill is designed to analyze all builds at once, meaning all possible
> architectures; it assumes that "koji download-build" fetches all built RPMs.
> This concept of instances breaks my assumptions. For now, we'll have to live
> with x86- and noarch-only analysis. The Taskotron effort should take this
> into consideration.
s/x86-/main arch-/

(In reply to Zbigniew Jędrzejewski-Szmek from comment #4)
> (In reply to Ed Santiago from comment #3)
> > Oh, yikes. This certainly makes things interesting! I had assumed that a
> > koji build notification meant "all builds on all architectures are done".
> AFAIU, build on those secondary architectures (or whatever the correct term
> is) are independent. I think that they are scheduled by some bot, possibly
> with a substantial delay. So I don't think that handling them together is
> possible.

Neither do I, the different koji instances for secondary arches really seem to be different systems with no connection except the dist-git.

> rpmgrill is designed to analyze all builds at once, meaning all possible
> architectures; it assumes that "koji download-build" fetches all built RPMs.

Actually, there are separated clients for secondary koji instances -- s390-koji, ppc-koji and arm-koji -- all packaged in fedora-packager component.

> This concept of instances breaks my assumptions. For now, we'll have to live
> with x86- and noarch-only analysis. The Taskotron effort should take this
> into consideration.

Yeah, we should think on that, but the reality with secondary arches in Fedora is that they usually get less care than primary arches, so IMHO we should be fine with checking only x86- in Taskotron at least for beginning.

This bug appears to have been reported against 'rawhide' during the Fedora 22 development cycle.
Changing version to '22'.

More information and reason for this action is here:
https://fedoraproject.org/wiki/Fedora_Program_Management/HouseKeeping/Fedora22

Fedora 22 changed to end-of-life (EOL) status on 2016-07-19. Fedora 22 is
no longer maintained, which means that it will not receive any further
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of
Fedora please feel free to reopen this bug against that version. If you
are unable to reopen this bug, please file a new report against the
current release. If you experience problems, please add a comment to this
bug.

Thank you for reporting this bug and we are sorry it could not be fixed.