Bug 1475867

Summary:	Running Jenkins Builds Write to API every second
Product:	OpenShift Container Platform	Reporter:	Eric Paris <eparis>
Component:	Build	Assignee:	Gabe Montero <gmontero>
Status:	CLOSED ERRATA	QA Contact:	Wenjing Zheng <wzheng>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	3.6.0	CC:	aos-bugs, bparees, cewong, gmontero, pweil, spadgett
Target Milestone:	---
Target Release:	3.7.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:	Cause: the openshift jenkins sync plugin was updating jenkins pipeline build status annotations every second, regardless of whether the status changed Consequence: the frequency of updates would put undo stress on the etcd instance backing openshift master Fix: jenkins pipeline build status annotations are only updated if the status actually changes, or 30 seconds have passed Result: openshift sync plugins is a more responsible player in an openshift cluster	Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-11-28 22:05:56 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Eric Paris 2017-07-27 13:47:29 UTC

oc v3.6.170

I see a jenkins build running which updates every second. We are updating the build's .metadata.annotations.openshift.io/jenkins-status-json

Inside that annotation it appears that queueDurationMillis and endTimeMillis are updated.

I'm going to attach the build json, build pod json, and build pod logs as attachments (easier to read and private)

Comment 4 Cesar Wong 2017-07-27 13:56:02 UTC

This is the sync plugin updating the status of the build as the pipeline progresses. My understanding is that this is working as designed, but will let Gabe comment since he's the expert.

Comment 5 Gabe Montero 2017-07-27 15:10:16 UTC

Sam - fyi - this is what I was pinging you about on freenode.

The sync plugin is currently updating the annotation for pipeline build status that the console consumes every second.  

That frequency of etcd updates is being called out here.

How bad of an impact would it be to the console's visualization of pipeline builds if that frequency was say every 10 seconds?  Or only updated as the build transitions from 1 stage to the next?

thanks,
gabe

Comment 6 Samuel Padgett 2017-07-27 15:15:08 UTC

It shouldn't be too bad, although we're currently using it for the duration that we show in the console pipeline visualization. So we're either need to show less granularity -- "less than a minute" instead of 17s -- or calculate the duration on the client. Either way I'm confident we can do something in the console if this changes.

Comment 7 Gabe Montero 2017-08-01 15:25:04 UTC

quick update ... v0.1.25 of the sync plugin is available in the update center and the centos openshift jenkins image will contain it by EOB today.

WRT rhel, per conversation I had with Justin Pierce, certainly it will be at least a week, probably more, before we could entertain building post 3.6 rhel images, given the post 3.6 GA settling that is still occurring.

I'll make a note to try and update this bug once 3.7 / 3.6.1 images start happening and if one of those has this fix.

Comment 8 Ben Parees 2017-08-24 16:05:58 UTC

PR: https://github.com/openshift/jenkins/pull/341

Comment 9 Wenjing Zheng 2017-09-15 06:57:45 UTC

Have done some regression test with sync plugin 0.1.26, no issues found, will verify this bug.

Comment 12 errata-xmlrpc 2017-11-28 22:05:56 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:3188