Bug 526275 - New package builds via koji-shadow force koji into odd state
Summary: New package builds via koji-shadow force koji into odd state
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: Fedora
Classification: Fedora
Component: koji
Version: 11
Hardware: All
OS: Linux
low
medium
Target Milestone: ---
Assignee: Mike McLean
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2009-09-29 15:54 UTC by Josh Boyer
Modified: 2013-01-10 05:29 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2010-06-28 14:50:53 UTC


Attachments (Terms of Use)
screencast ogg/theora (968.57 KB, application/octet-stream)
2009-09-29 16:26 UTC, Josh Boyer
no flags Details
patch to avoid the traceback (1.15 KB, patch)
2009-10-01 21:37 UTC, Mike McLean
no flags Details | Diff

Description Josh Boyer 2009-09-29 15:54:03 UTC
Description of problem:

When doing a new package build via koji-shadow the builds tend to succeed, however the overall build tasks fail with:

<type 'exceptions.KeyError'>: 9091

The individual build subtasks are clearly marked as fail, however koji thinks
the overall build task is still in 'building' state.

How reproducible:

Fairly reliable.

Steps to Reproduce:
1. Run owner-sync-pkgdb dist-f12 to sync local koji with new packages
2. Run koji-shadow and wait for one to get built
3. Stare at the confusion
  
Actual results:

Koji fails with the error above and fails to put the build task in Failed state

Expected results:

Well, either successfully completing the build (optimal) or at least being consistent in task state.

Additional info:

The only way I've been able to work around these is to actually delete the build from the koji db via psql commands, remove any directories in /mnt/koji/packages/<package>/ and then manually submit the build via 'koji build dist-f12 <package>'.  Assuming there are no build failures, the manual invocation actually works fine.

Comment 1 Mike McLean 2009-09-29 16:13:05 UTC
Can you provide a full traceback?
Where exactly did this KeyError message appear? (e.g. on the command line, task display in the ui, logs)
Did this occur on a koji server with a web ui that I can access? If so, please send me a link to the task.

Comment 2 Josh Boyer 2009-09-29 16:25:46 UTC
(In reply to comment #1)
> Can you provide a full traceback?

I see no traceback.  Where would it be?

> Where exactly did this KeyError message appear? (e.g. on the command line, task
> display in the ui, logs)

display in the ui.

> Did this occur on a koji server with a web ui that I can access? If so, please
> send me a link to the task.  

My ISP sucks and blocks access to port 80.  As crappy as it may be, I'll attach a screencast of what I'm talking about and we can go from there.

Comment 3 Josh Boyer 2009-09-29 16:26:44 UTC
Created attachment 363040 [details]
screencast ogg/theora

Comment 4 Mike McLean 2009-09-29 16:41:50 UTC
Look in kojid.log on the builder that had the failed build task. Search from the end for the task id that failed (2782), and/or search for the error message you saw. Hopefully there will be a full traceback near there.

Comment 5 Josh Boyer 2009-09-29 16:46:27 UTC
The log has been rotated out, but here is a similar KeyError and backtrace:

2009-09-29 04:36:46,699 [INFO] koji.build.TaskManager: Attempting to take task 2997
2009-09-29 04:36:46,883 [WARNING] koji.build.TaskManager: FAULT:
Traceback (most recent call last):
  File "/usr/sbin/kojid", line 1275, in runTask
    response = (handler.run(),)
  File "/usr/sbin/kojid", line 1351, in run
    return self.handler(*self.params,**self.opts)
  File "/usr/sbin/kojid", line 1782, in handler
    session.host.failBuild(self.id, build_id)
  File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1255, in __call__
    return self.__func(self.__name,args,opts)
  File "/usr/lib/python2.6/site-packages/koji/__init__.py", line 1501, in _callMethod
    raise err
Fault: <Fault 1: "<type 'exceptions.KeyError'>: 9117">

Comment 6 Mike McLean 2009-09-29 17:31:14 UTC
Ok, the error we see is happening when the host attempts to fail the build (inside of an exception handler). Since the subtasks succeeded, I think the failure is caused because the builder is getting an earlier error during the completeBuild call. They may have a common cause.

Can you look in /var/log/httpd/error_log on your hub and see if you can find both tracebacks?

Comment 7 Josh Boyer 2009-09-29 18:16:56 UTC
I had debug enabled, so here is what I see for the 9117 KeyError in error_log:

    SELECT package.id, package.name, tag.id, tag.name, users.id, users.name, extra_arches, tag_packages.blocked
    FROM tag_packages
    JOIN tag on tag.id = tag_packages.tag_id
    JOIN package ON package.id = tag_packages.package_id
    JOIN users ON users.id = tag_packages.owner
    WHERE (active = TRUE)
        AND tag.id = 1
            AND package.id = 9117
Execute operation completed in 0.0039 seconds
fetchall operation completed in 0.0000 seconds
SELECT parent_id,name,priority,maxdepth,intransitive,noconfig,pkg_filter FROM tag_inheritance JOIN tag ON parent_id = id
    WHERE (active = TRUE) AND tag_id = 1
    ORDER BY priority
    
Execute operation completed in 0.0520 seconds
fetchall operation completed in 0.0000 seconds
Traceback (most recent call last):
  File "/usr/share/koji-hub/kojixmlrpc.py", line 199, in _marshaled_dispatch
    response = self._dispatch(method, params)
  File "/usr/share/koji-hub/kojixmlrpc.py", line 268, in _dispatch
    ret = func(*params,**opts)
  File "/usr/share/koji-hub/kojihub.py", line 7126, in failBuild
    build_notification(task_id, build_id)
  File "/usr/share/koji-hub/kojihub.py", line 3985, in build_notification
    recipients = get_notification_recipients(build, dest_tag, build['state'])
  File "/usr/share/koji-hub/kojihub.py", line 3937, in get_notification_recipien
ts
    emails.append('%s@%s' % (packages[package_id]['owner_name'], email_domain))
KeyError: 9117

Returning 295 bytes after 0.338667 seconds


(I have no turned off debug because error_log is 5GiB!)

Comment 8 Mike McLean 2009-09-29 19:48:26 UTC
Ok, I think I see the problem. koji-shadow does some pretty unusual tag juggling to accomplish its goals and by the time it gets to doing notifications the tag date may have changed dramatically.

The easy workaround is to disable notifications on the hub by setting DisableNotifications to true in hub.conf.

Otherwise I should have a patch for this issue shortly.

Comment 9 Josh Boyer 2009-09-29 20:37:44 UTC
(In reply to comment #8)
> Ok, I think I see the problem. koji-shadow does some pretty unusual tag
> juggling to accomplish its goals and by the time it gets to doing notifications
> the tag date may have changed dramatically.
> 
> The easy workaround is to disable notifications on the hub by setting
> DisableNotifications to true in hub.conf.

Oh, sweet.  I've been wanting to do that anyway.

> Otherwise I should have a patch for this issue shortly.  

Be happy to test when you do.

Comment 10 Mike McLean 2009-10-01 21:37:29 UTC
Created attachment 363399 [details]
patch to avoid the traceback

This patch to kojihub.py should resolve the issue.

Comment 11 Bug Zapper 2010-04-28 10:36:56 UTC
This message is a reminder that Fedora 11 is nearing its end of life.
Approximately 30 (thirty) days from now Fedora will stop maintaining
and issuing updates for Fedora 11.  It is Fedora's policy to close all
bug reports from releases that are no longer maintained.  At that time
this bug will be closed as WONTFIX if it remains open with a Fedora 
'version' of '11'.

Package Maintainer: If you wish for this bug to remain open because you
plan to fix it in a currently maintained version, simply change the 'version' 
to a later Fedora version prior to Fedora 11's end of life.

Bug Reporter: Thank you for reporting this issue and we are sorry that 
we may not be able to fix it before Fedora 11 is end of life.  If you 
would still like to see this bug fixed and are able to reproduce it 
against a later version of Fedora please change the 'version' of this 
bug to the applicable version.  If you are unable to change the version, 
please add a comment here and someone will do it for you.

Although we aim to fix as many bugs as possible during every release's 
lifetime, sometimes those efforts are overtaken by events.  Often a 
more recent Fedora release includes newer upstream software that fixes 
bugs or makes them obsolete.

The process we are following is described here: 
http://fedoraproject.org/wiki/BugZappers/HouseKeeping

Comment 12 Bug Zapper 2010-06-28 14:50:53 UTC
Fedora 11 changed to end-of-life (EOL) status on 2010-06-25. Fedora 11 is 
no longer maintained, which means that it will not receive any further 
security or bug fix updates. As a result we are closing this bug.

If you can reproduce this bug against a currently maintained version of 
Fedora please feel free to reopen this bug against that version.

Thank you for reporting this bug and we are sorry it could not be fixed.


Note You need to log in before you can comment on or make changes to this bug.