Bug 817625 - darkserver adds significant time to builds with large debuginfo packages
darkserver adds significant time to builds with large debuginfo packages
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: darkserver (Show other bugs)
17
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: kushaldas@gmail.com
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-04-30 12:43 EDT by Josh Boyer
Modified: 2012-06-30 21:36 EDT (History)
9 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-06-30 21:36:02 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Josh Boyer 2012-04-30 12:43:35 EDT
Description of problem:

I recently noticed that the Fedora kernel builds were taking between 30 and 60 minutes longer than usual.  I watched a build and noticed that this time was added _after_ all of the build tasks had completed.  That is to say, the packages were all successfully built and those tasks done, but the overall build task was off doing something that was neither logged nor visible from the web UI.

A discussion with Kevin Fenzi from infrastructure pointed out that the darkserver plugin is now running on the Fedora koji instance, and that it goes and scrapes information from all of the debuginfo packages built during a build before the build task is marked as complete.  That is likely a minimal impact for most packages, however the Fedora kernel package has a number of relatively large debuginfo packages per build.  It seems going through all of those is adding significant build time.

Version-Release number of selected component (if applicable):


How reproducible:

Always


Steps to Reproduce:
1.  Start a kernel build
2.  Wait for all build tasks to finish
3.  Watch the overall koji task continue to sit there doing nothing for a while
  
Actual results:

Significant extra time spent per kernel build

Expected results:

Not sure.  Not having to wait that long I suppose.

Additional info:

Kevin sent an email to Kushal pointing to this issue, but we haven't seen any activity.

Perhaps the koji plugin could be dropped, and darkserver could instead run asynchronously after a build completes.  Maybe even tied into the Fedora message bus.
Comment 1 kushaldas@gmail.com 2012-05-02 08:21:16 EDT
I started looking for a possible solution for this after Kevin informed me about the issue. I also asked Dennis Gilmore if we can run the plugin in an asynchronous way, but still now can not find any way to achieve that.

The reason behind the extra time is due to large number of ELF files in the kernel package. Even in my test koji setup all tests took that kind of extra time.

For now I will make another release which will skip the kernel packages (in koji-plugin) and will keep working find a better way to do async import of the data.
Comment 2 Fedora Update System 2012-05-02 09:04:58 EDT
darkserver-0.5.4-1.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/darkserver-0.5.4-1.el6
Comment 3 Kevin Fenzi 2012-05-03 15:36:41 EDT
We can apply this for now to help out the kernel team, but not sure it's a good long term solution... 

There are probibly other packages that are large and now have an extra time to finish builds also. 

Perhaps things could move to a model where the plugin just notifies another process to extract buildid's from a queue of packages or something?
Comment 4 Fedora Update System 2012-05-30 21:11:40 EDT
Package darkserver-0.5.4-1.el6:
* should fix your issue,
* was pushed to the Fedora EPEL 6 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=epel-testing darkserver-0.5.4-1.el6'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-EPEL-2012-6006/darkserver-0.5.4-1.el6
then log in and leave karma (feedback).
Comment 5 Rex Dieter 2012-06-08 14:04:58 EDT
seems kdelibs may be affected by this too.  for example,
http://koji.fedoraproject.org/koji/buildinfo?buildID=323085
builds finished at 14:57 and it's 18:03:26 now, and the parent task is still going.
Comment 6 Kevin Kofler 2012-06-08 16:38:37 EDT
IMHO, darkserver should be disabled until this is sorted out.
Comment 7 Kushal Das 2012-06-09 01:09:14 EDT
Go ahead and disable it. I already found a way to import the missing builds later.
Comment 8 Kevin Fenzi 2012-06-09 14:05:55 EDT
ok. If it looks like there's a good time this weekend I'll disable it then... otherwise we have a outage next wed, and we can just do it then.
Comment 9 Kevin Fenzi 2012-06-10 00:43:20 EDT
I've just disabled it.
Comment 10 Fedora Update System 2012-06-30 21:36:02 EDT
darkserver-0.5.4-1.el6 has been pushed to the Fedora EPEL 6 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.