Description of problem: When doing a new package build via koji-shadow the builds tend to succeed, however the overall build tasks fail with: <type 'exceptions.KeyError'>: 9091 The individual build subtasks are clearly marked as fail, however koji thinks the overall build task is still in 'building' state. How reproducible: Fairly reliable. Steps to Reproduce: 1. Run owner-sync-pkgdb dist-f12 to sync local koji with new packages 2. Run koji-shadow and wait for one to get built 3. Stare at the confusion Actual results: Koji fails with the error above and fails to put the build task in Failed state Expected results: Well, either successfully completing the build (optimal) or at least being consistent in task state. Additional info: The only way I've been able to work around these is to actually delete the build from the koji db via psql commands, remove any directories in /mnt/koji/packages/<package>/ and then manually submit the build via 'koji build dist-f12 <package>'. Assuming there are no build failures, the manual invocation actually works fine.
Can you provide a full traceback? Where exactly did this KeyError message appear? (e.g. on the command line, task display in the ui, logs) Did this occur on a koji server with a web ui that I can access? If so, please send me a link to the task.
(In reply to comment #1) > Can you provide a full traceback? I see no traceback. Where would it be? > Where exactly did this KeyError message appear? (e.g. on the command line, task > display in the ui, logs) display in the ui. > Did this occur on a koji server with a web ui that I can access? If so, please > send me a link to the task. My ISP sucks and blocks access to port 80. As crappy as it may be, I'll attach a screencast of what I'm talking about and we can go from there.
Created attachment 363040 [details] screencast ogg/theora
Look in kojid.log on the builder that had the failed build task. Search from the end for the task id that failed (2782), and/or search for the error message you saw. Hopefully there will be a full traceback near there.
The log has been rotated out, but here is a similar KeyError and backtrace: 2009-09-29 04:36:46,699 [INFO] koji.build.TaskManager: Attempting to take task 2997 2009-09-29 04:36:46,883 [WARNING] koji.build.TaskManager: FAULT: Traceback (most recent call last): File "/usr/sbin/kojid", line 1275, in runTask response = (handler.run(),) File "/usr/sbin/kojid", line 1351, in run return self.handler(*self.params,**self.opts) File "/usr/sbin/kojid", line 1782, in handler session.host.failBuild(self.id, build_id) File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1255, in __call__ return self.__func(self.__name,args,opts) File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1501, in _callMethod raise err Fault: <Fault 1: "<type 'exceptions.KeyError'>: 9117">
Ok, the error we see is happening when the host attempts to fail the build (inside of an exception handler). Since the subtasks succeeded, I think the failure is caused because the builder is getting an earlier error during the completeBuild call. They may have a common cause. Can you look in /var/log/httpd/error_log on your hub and see if you can find both tracebacks?
I had debug enabled, so here is what I see for the 9117 KeyError in error_log: SELECT package.id, package.name, tag.id, tag.name, users.id, users.name, extra_arches, tag_packages.blocked FROM tag_packages JOIN tag on tag.id = tag_packages.tag_id JOIN package ON package.id = tag_packages.package_id JOIN users ON users.id = tag_packages.owner WHERE (active = TRUE) AND tag.id = 1 AND package.id = 9117 Execute operation completed in 0.0039 seconds fetchall operation completed in 0.0000 seconds SELECT parent_id,name,priority,maxdepth,intransitive,noconfig,pkg_filter FROM tag_inheritance JOIN tag ON parent_id = id WHERE (active = TRUE) AND tag_id = 1 ORDER BY priority Execute operation completed in 0.0520 seconds fetchall operation completed in 0.0000 seconds Traceback (most recent call last): File "/usr/share/koji-hub/kojixmlrpc.py", line 199, in _marshaled_dispatch response = self._dispatch(method, params) File "/usr/share/koji-hub/kojixmlrpc.py", line 268, in _dispatch ret = func(*params,**opts) File "/usr/share/koji-hub/kojihub.py", line 7126, in failBuild build_notification(task_id, build_id) File "/usr/share/koji-hub/kojihub.py", line 3985, in build_notification recipients = get_notification_recipients(build, dest_tag, build['state']) File "/usr/share/koji-hub/kojihub.py", line 3937, in get_notification_recipien ts emails.append('%s@%s' % (packages[package_id]['owner_name'], email_domain)) KeyError: 9117 Returning 295 bytes after 0.338667 seconds (I have no turned off debug because error_log is 5GiB!)
Ok, I think I see the problem. koji-shadow does some pretty unusual tag juggling to accomplish its goals and by the time it gets to doing notifications the tag date may have changed dramatically. The easy workaround is to disable notifications on the hub by setting DisableNotifications to true in hub.conf. Otherwise I should have a patch for this issue shortly.
(In reply to comment #8) > Ok, I think I see the problem. koji-shadow does some pretty unusual tag > juggling to accomplish its goals and by the time it gets to doing notifications > the tag date may have changed dramatically. > > The easy workaround is to disable notifications on the hub by setting > DisableNotifications to true in hub.conf. Oh, sweet. I've been wanting to do that anyway. > Otherwise I should have a patch for this issue shortly. Be happy to test when you do.
Created attachment 363399 [details] patch to avoid the traceback This patch to kojihub.py should resolve the issue.
This message is a reminder that Fedora 11 is nearing its end of life. Approximately 30 (thirty) days from now Fedora will stop maintaining and issuing updates for Fedora 11. It is Fedora's policy to close all bug reports from releases that are no longer maintained. At that time this bug will be closed as WONTFIX if it remains open with a Fedora 'version' of '11'. Package Maintainer: If you wish for this bug to remain open because you plan to fix it in a currently maintained version, simply change the 'version' to a later Fedora version prior to Fedora 11's end of life. Bug Reporter: Thank you for reporting this issue and we are sorry that we may not be able to fix it before Fedora 11 is end of life. If you would still like to see this bug fixed and are able to reproduce it against a later version of Fedora please change the 'version' of this bug to the applicable version. If you are unable to change the version, please add a comment here and someone will do it for you. Although we aim to fix as many bugs as possible during every release's lifetime, sometimes those efforts are overtaken by events. Often a more recent Fedora release includes newer upstream software that fixes bugs or makes them obsolete. The process we are following is described here: http://fedoraproject.org/wiki/BugZappers/HouseKeeping
Fedora 11 changed to end-of-life (EOL) status on 2010-06-25. Fedora 11 is no longer maintained, which means that it will not receive any further security or bug fix updates. As a result we are closing this bug. If you can reproduce this bug against a currently maintained version of Fedora please feel free to reopen this bug against that version. Thank you for reporting this bug and we are sorry it could not be fixed.