|Summary:||Running Jenkins Builds Write to API every second|
|Product:||OpenShift Container Platform||Reporter:||Eric Paris <eparis>|
|Component:||Build||Assignee:||Gabe Montero <gmontero>|
|Status:||CLOSED ERRATA||QA Contact:||Wenjing Zheng <wzheng>|
|Version:||3.6.0||CC:||aos-bugs, bparees, cewong, gmontero, pweil, spadgett|
|Fixed In Version:||Doc Type:||Bug Fix|
Cause: the openshift jenkins sync plugin was updating jenkins pipeline build status annotations every second, regardless of whether the status changed Consequence: the frequency of updates would put undo stress on the etcd instance backing openshift master Fix: jenkins pipeline build status annotations are only updated if the status actually changes, or 30 seconds have passed Result: openshift sync plugins is a more responsible player in an openshift cluster
|Last Closed:||2017-11-28 22:05:56 UTC||Type:||Bug|
|oVirt Team:||---||RHEL 7.3 requirements from Atomic Host:|
|Cloudforms Team:||---||Target Upstream Version:|
Description Eric Paris 2017-07-27 13:47:29 UTC
oc v3.6.170 I see a jenkins build running which updates every second. We are updating the build's .metadata.annotations.openshift.io/jenkins-status-json Inside that annotation it appears that queueDurationMillis and endTimeMillis are updated. I'm going to attach the build json, build pod json, and build pod logs as attachments (easier to read and private)
Comment 4 Cesar Wong 2017-07-27 13:56:02 UTC
This is the sync plugin updating the status of the build as the pipeline progresses. My understanding is that this is working as designed, but will let Gabe comment since he's the expert.
Comment 5 Gabe Montero 2017-07-27 15:10:16 UTC
Sam - fyi - this is what I was pinging you about on freenode. The sync plugin is currently updating the annotation for pipeline build status that the console consumes every second. That frequency of etcd updates is being called out here. How bad of an impact would it be to the console's visualization of pipeline builds if that frequency was say every 10 seconds? Or only updated as the build transitions from 1 stage to the next? thanks, gabe
Comment 6 Samuel Padgett 2017-07-27 15:15:08 UTC
It shouldn't be too bad, although we're currently using it for the duration that we show in the console pipeline visualization. So we're either need to show less granularity -- "less than a minute" instead of 17s -- or calculate the duration on the client. Either way I'm confident we can do something in the console if this changes.
Comment 7 Gabe Montero 2017-08-01 15:25:04 UTC
quick update ... v0.1.25 of the sync plugin is available in the update center and the centos openshift jenkins image will contain it by EOB today. WRT rhel, per conversation I had with Justin Pierce, certainly it will be at least a week, probably more, before we could entertain building post 3.6 rhel images, given the post 3.6 GA settling that is still occurring. I'll make a note to try and update this bug once 3.7 / 3.6.1 images start happening and if one of those has this fix.
Comment 9 Wenjing Zheng 2017-09-15 06:57:45 UTC
Have done some regression test with sync plugin 0.1.26, no issues found, will verify this bug.
Comment 12 errata-xmlrpc 2017-11-28 22:05:56 UTC
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2017:3188