Bug 1475867 - Running Jenkins Builds Write to API every second
Running Jenkins Builds Write to API every second
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build (Show other bugs)
Unspecified Unspecified
unspecified Severity unspecified
: ---
: 3.7.0
Assigned To: Gabe Montero
Wenjing Zheng
Depends On:
  Show dependency treegraph
Reported: 2017-07-27 09:47 EDT by Eric Paris
Modified: 2017-11-28 17:05 EST (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: the openshift jenkins sync plugin was updating jenkins pipeline build status annotations every second, regardless of whether the status changed Consequence: the frequency of updates would put undo stress on the etcd instance backing openshift master Fix: jenkins pipeline build status annotations are only updated if the status actually changes, or 30 seconds have passed Result: openshift sync plugins is a more responsible player in an openshift cluster
Story Points: ---
Clone Of:
Last Closed: 2017-11-28 17:05:56 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Eric Paris 2017-07-27 09:47:29 EDT
oc v3.6.170

I see a jenkins build running which updates every second. We are updating the build's .metadata.annotations.openshift.io/jenkins-status-json

Inside that annotation it appears that queueDurationMillis and endTimeMillis are updated.

I'm going to attach the build json, build pod json, and build pod logs as attachments (easier to read and private)
Comment 4 Cesar Wong 2017-07-27 09:56:02 EDT
This is the sync plugin updating the status of the build as the pipeline progresses. My understanding is that this is working as designed, but will let Gabe comment since he's the expert.
Comment 5 Gabe Montero 2017-07-27 11:10:16 EDT
Sam - fyi - this is what I was pinging you about on freenode.

The sync plugin is currently updating the annotation for pipeline build status that the console consumes every second.  

That frequency of etcd updates is being called out here.

How bad of an impact would it be to the console's visualization of pipeline builds if that frequency was say every 10 seconds?  Or only updated as the build transitions from 1 stage to the next?

Comment 6 Samuel Padgett 2017-07-27 11:15:08 EDT
It shouldn't be too bad, although we're currently using it for the duration that we show in the console pipeline visualization. So we're either need to show less granularity -- "less than a minute" instead of 17s -- or calculate the duration on the client. Either way I'm confident we can do something in the console if this changes.
Comment 7 Gabe Montero 2017-08-01 11:25:04 EDT
quick update ... v0.1.25 of the sync plugin is available in the update center and the centos openshift jenkins image will contain it by EOB today.

WRT rhel, per conversation I had with Justin Pierce, certainly it will be at least a week, probably more, before we could entertain building post 3.6 rhel images, given the post 3.6 GA settling that is still occurring.

I'll make a note to try and update this bug once 3.7 / 3.6.1 images start happening and if one of those has this fix.
Comment 8 Ben Parees 2017-08-24 12:05:58 EDT
PR: https://github.com/openshift/jenkins/pull/341
Comment 9 Wenjing Zheng 2017-09-15 02:57:45 EDT
Have done some regression test with sync plugin 0.1.26, no issues found, will verify this bug.
Comment 12 errata-xmlrpc 2017-11-28 17:05:56 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.