Red Hat Bugzilla – Bug 817625
darkserver adds significant time to builds with large debuginfo packages
Last modified: 2012-06-30 21:36:02 EDT
Description of problem:
I recently noticed that the Fedora kernel builds were taking between 30 and 60 minutes longer than usual. I watched a build and noticed that this time was added _after_ all of the build tasks had completed. That is to say, the packages were all successfully built and those tasks done, but the overall build task was off doing something that was neither logged nor visible from the web UI.
A discussion with Kevin Fenzi from infrastructure pointed out that the darkserver plugin is now running on the Fedora koji instance, and that it goes and scrapes information from all of the debuginfo packages built during a build before the build task is marked as complete. That is likely a minimal impact for most packages, however the Fedora kernel package has a number of relatively large debuginfo packages per build. It seems going through all of those is adding significant build time.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1. Start a kernel build
2. Wait for all build tasks to finish
3. Watch the overall koji task continue to sit there doing nothing for a while
Significant extra time spent per kernel build
Not sure. Not having to wait that long I suppose.
Kevin sent an email to Kushal pointing to this issue, but we haven't seen any activity.
Perhaps the koji plugin could be dropped, and darkserver could instead run asynchronously after a build completes. Maybe even tied into the Fedora message bus.
I started looking for a possible solution for this after Kevin informed me about the issue. I also asked Dennis Gilmore if we can run the plugin in an asynchronous way, but still now can not find any way to achieve that.
The reason behind the extra time is due to large number of ELF files in the kernel package. Even in my test koji setup all tests took that kind of extra time.
For now I will make another release which will skip the kernel packages (in koji-plugin) and will keep working find a better way to do async import of the data.
darkserver-0.5.4-1.el6 has been submitted as an update for Fedora EPEL 6.
We can apply this for now to help out the kernel team, but not sure it's a good long term solution...
There are probibly other packages that are large and now have an extra time to finish builds also.
Perhaps things could move to a model where the plugin just notifies another process to extract buildid's from a queue of packages or something?
* should fix your issue,
* was pushed to the Fedora EPEL 6 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=epel-testing darkserver-0.5.4-1.el6'
as soon as you are able to.
Please go to the following url:
then log in and leave karma (feedback).
seems kdelibs may be affected by this too. for example,
builds finished at 14:57 and it's 18:03:26 now, and the parent task is still going.
IMHO, darkserver should be disabled until this is sorted out.
Go ahead and disable it. I already found a way to import the missing builds later.
ok. If it looks like there's a good time this weekend I'll disable it then... otherwise we have a outage next wed, and we can just do it then.
I've just disabled it.
darkserver-0.5.4-1.el6 has been pushed to the Fedora EPEL 6 stable repository. If problems still persist, please make note of it in this bug report.