Bug 1475867 - Running Jenkins Builds Write to API every second
Summary: Running Jenkins Builds Write to API every second
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 3.6.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 3.7.0
Assignee: Gabe Montero
QA Contact: Wenjing Zheng
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-07-27 13:47 UTC by Eric Paris
Modified: 2017-11-28 22:05 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: the openshift jenkins sync plugin was updating jenkins pipeline build status annotations every second, regardless of whether the status changed Consequence: the frequency of updates would put undo stress on the etcd instance backing openshift master Fix: jenkins pipeline build status annotations are only updated if the status actually changes, or 30 seconds have passed Result: openshift sync plugins is a more responsible player in an openshift cluster
Clone Of:
Environment:
Last Closed: 2017-11-28 22:05:56 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2017:3188 normal SHIPPED_LIVE Moderate: Red Hat OpenShift Container Platform 3.7 security, bug, and enhancement update 2017-11-29 02:34:54 UTC

Description Eric Paris 2017-07-27 13:47:29 UTC
oc v3.6.170

I see a jenkins build running which updates every second. We are updating the build's .metadata.annotations.openshift.io/jenkins-status-json

Inside that annotation it appears that queueDurationMillis and endTimeMillis are updated.

I'm going to attach the build json, build pod json, and build pod logs as attachments (easier to read and private)

Comment 4 Cesar Wong 2017-07-27 13:56:02 UTC
This is the sync plugin updating the status of the build as the pipeline progresses. My understanding is that this is working as designed, but will let Gabe comment since he's the expert.

Comment 5 Gabe Montero 2017-07-27 15:10:16 UTC
Sam - fyi - this is what I was pinging you about on freenode.

The sync plugin is currently updating the annotation for pipeline build status that the console consumes every second.  

That frequency of etcd updates is being called out here.

How bad of an impact would it be to the console's visualization of pipeline builds if that frequency was say every 10 seconds?  Or only updated as the build transitions from 1 stage to the next?

thanks,
gabe

Comment 6 Samuel Padgett 2017-07-27 15:15:08 UTC
It shouldn't be too bad, although we're currently using it for the duration that we show in the console pipeline visualization. So we're either need to show less granularity -- "less than a minute" instead of 17s -- or calculate the duration on the client. Either way I'm confident we can do something in the console if this changes.

Comment 7 Gabe Montero 2017-08-01 15:25:04 UTC
quick update ... v0.1.25 of the sync plugin is available in the update center and the centos openshift jenkins image will contain it by EOB today.

WRT rhel, per conversation I had with Justin Pierce, certainly it will be at least a week, probably more, before we could entertain building post 3.6 rhel images, given the post 3.6 GA settling that is still occurring.

I'll make a note to try and update this bug once 3.7 / 3.6.1 images start happening and if one of those has this fix.

Comment 8 Ben Parees 2017-08-24 16:05:58 UTC
PR: https://github.com/openshift/jenkins/pull/341

Comment 9 Wenjing Zheng 2017-09-15 06:57:45 UTC
Have done some regression test with sync plugin 0.1.26, no issues found, will verify this bug.

Comment 12 errata-xmlrpc 2017-11-28 22:05:56 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188


Note You need to log in before you can comment on or make changes to this bug.