Bug 817625

Summary:	darkserver adds significant time to builds with large debuginfo packages
Product:	[Fedora] Fedora	Reporter:	Josh Boyer <jwboyer>
Component:	darkserver	Assignee:	kushaldas@gmail.com <mail>
Status:	CLOSED ERRATA	QA Contact:	Fedora Extras Quality Assurance <extras-qa>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	17	CC:	alekcejk, jreznik, kdas, kevin, kevin, mail, metherid, michel, rdieter
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2012-07-01 01:36:02 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Josh Boyer 2012-04-30 16:43:35 UTC

Description of problem:

I recently noticed that the Fedora kernel builds were taking between 30 and 60 minutes longer than usual. I watched a build and noticed that this time was added _after_ all of the build tasks had completed. That is to say, the packages were all successfully built and those tasks done, but the overall build task was off doing something that was neither logged nor visible from the web UI.

A discussion with Kevin Fenzi from infrastructure pointed out that the darkserver plugin is now running on the Fedora koji instance, and that it goes and scrapes information from all of the debuginfo packages built during a build before the build task is marked as complete. That is likely a minimal impact for most packages, however the Fedora kernel package has a number of relatively large debuginfo packages per build. It seems going through all of those is adding significant build time.

Version-Release number of selected component (if applicable):

How reproducible:

Always

Steps to Reproduce:
1. Start a kernel build
2. Wait for all build tasks to finish
3. Watch the overall koji task continue to sit there doing nothing for a while

Actual results:

Significant extra time spent per kernel build

Expected results:

Not sure. Not having to wait that long I suppose.

Additional info:

Kevin sent an email to Kushal pointing to this issue, but we haven't seen any activity.

Perhaps the koji plugin could be dropped, and darkserver could instead run asynchronously after a build completes. Maybe even tied into the Fedora message bus.

Comment 1 kushaldas@gmail.com 2012-05-02 12:21:16 UTC

I started looking for a possible solution for this after Kevin informed me about the issue. I also asked Dennis Gilmore if we can run the plugin in an asynchronous way, but still now can not find any way to achieve that.

The reason behind the extra time is due to large number of ELF files in the kernel package. Even in my test koji setup all tests took that kind of extra time.

For now I will make another release which will skip the kernel packages (in koji-plugin) and will keep working find a better way to do async import of the data.

Comment 2 Fedora Update System 2012-05-02 13:04:58 UTC

darkserver-0.5.4-1.el6 has been submitted as an update for Fedora EPEL 6.
https://admin.fedoraproject.org/updates/darkserver-0.5.4-1.el6

Comment 3 Kevin Fenzi 2012-05-03 19:36:41 UTC

We can apply this for now to help out the kernel team, but not sure it's a good long term solution... 

There are probibly other packages that are large and now have an extra time to finish builds also. 

Perhaps things could move to a model where the plugin just notifies another process to extract buildid's from a queue of packages or something?

Comment 4 Fedora Update System 2012-05-31 01:11:40 UTC

Package darkserver-0.5.4-1.el6:
* should fix your issue,
* was pushed to the Fedora EPEL 6 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=epel-testing darkserver-0.5.4-1.el6'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-EPEL-2012-6006/darkserver-0.5.4-1.el6
then log in and leave karma (feedback).

Comment 5 Rex Dieter 2012-06-08 18:04:58 UTC

seems kdelibs may be affected by this too.  for example,
http://koji.fedoraproject.org/koji/buildinfo?buildID=323085
builds finished at 14:57 and it's 18:03:26 now, and the parent task is still going.

Comment 6 Kevin Kofler 2012-06-08 20:38:37 UTC

IMHO, darkserver should be disabled until this is sorted out.

Comment 7 Kushal Das 2012-06-09 05:09:14 UTC

Go ahead and disable it. I already found a way to import the missing builds later.

Comment 8 Kevin Fenzi 2012-06-09 18:05:55 UTC

ok. If it looks like there's a good time this weekend I'll disable it then... otherwise we have a outage next wed, and we can just do it then.

Comment 9 Kevin Fenzi 2012-06-10 04:43:20 UTC

I've just disabled it.

Comment 10 Fedora Update System 2012-07-01 01:36:02 UTC

darkserver-0.5.4-1.el6 has been pushed to the Fedora EPEL 6 stable repository.  If problems still persist, please make note of it in this bug report.