Bug 1364948 - Builds don't sync consistently from openshift client with jenkins webconsole after jenkins server run more than 24 hours
Summary: Builds don't sync consistently from openshift client with jenkins webconsole ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: ImageStreams
Version: 3.3.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Jimmi Dyson
QA Contact: Wang Haoran
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-08 09:31 UTC by XiuJuan Wang
Modified: 2017-03-08 18:43 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Jenkins Sync plugin fails to consistently sync build changes from the OpenShift cluster Consequence: Builds created in OpenShift are not observed and executed by the Jenkins server Fix: Make sync logic more robust to ensure changes are not missed Result: Builds will be properly processed by the sync plugin and executed in Jenkins
Clone Of:
Environment:
Last Closed: 2017-01-18 12:52:04 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:0066 0 normal SHIPPED_LIVE Red Hat OpenShift Container Platform 3.4 RPM Release Advisory 2017-01-18 17:23:26 UTC

Description XiuJuan Wang 2016-08-08 09:31:22 UTC
Description of problem:
Created two pipeline apps,one slave uses maven image, one slave uses nodejs image.

After jenkins server run more than 24 hours,the synchronization doesn't work well.

1.The build numbers are not consistent.
maven slave pipeline app shows bc have built 11 times in openshift client, but there is 13 build in jenkins webconsole.
nodejs slave pipleline app shows bc have built 10 times in openshift client, but there is 12 builds in jenkins webcosnole.
Some builds are triggered from client, some from jenkins webconsole.

See the attachment.

2.The first new triggered build keep new status in both apps, no matter this build is complete/failed/cancelled in jenkins webconsole.

3.New triggered builds become to complete/failed/cancelled directly from new status(except the first new build), no running staus in openshift client.

New created pipeline apps work well.

Version-Release number of selected component (if applicable):
ci.dev.../openshift/jenkins-slave-nodejs-rhel7 965eb5e5766a
ci.dev.../openshift/jenkins-slave-maven-rhel7  7c6bbfb9656e
ci.dev.../openshift/jenkins-1-rhel7 bd3ab5d39d48

openshift v3.3.0.16
kubernetes v1.3.0+507d3a7

How reproducible:
always

Steps to Reproduce:
1.Created two pipeline apps,one slave uses maven image, one slave uses nodejs image.
2.Trigger some builds from openshift client and jenkins wenconsole
3.Leave jenkins server run more than 24 hours
4.Trigger some builds from openshift client and jenkins webconsole again
5.Check builds from openshift client and jenkins webconsole.

Actual results:
Builds can't sync well after jenkins server run long time.

Expected results:
Builds should sync well after jenkins server run long time.

Additional info:
There are errors in jenkins server pod log

INFO: onCompleted job/sample-pipeline/1/
Aug 08, 2016 4:18:03 AM jenkins.slaves.JnlpSlaveAgentProtocol$Handler$1 onClosed
WARNING: Computer.threadPoolForRemoting [#19] for afb6aa4b5300 terminated
java.io.EOFException
	at org.jenkinsci.remoting.nio.NioChannelHub$3.run(NioChannelHub.java:614)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112)
	at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Aug 08, 2016 4:18:04 AM io.fabric8.jenkins.openshiftsync.BuildSyncRunListener upsertBuild
INFO: Patching build in namespace xiuwang91 with name: sample-pipeline-1 phase: Failed
Aug 08, 2016 4:18:04 AM io.fabric8.jenkins.openshiftsync.BuildSyncRunListener onFinalized
INFO: onFinalized job/sample-pipeline/1/

Comment 1 Ben Parees 2016-08-16 13:34:16 UTC
tracking as github issue: https://github.com/fabric8io/openshift-jenkins-sync-plugin/issues/98

Comment 2 Jimmi Dyson 2016-09-23 14:49:44 UTC
I've released openshift-jenkins-sync-plugin 0.0.13 (https://repo.jenkins-ci.org/releases/io/fabric8/jenkins/plugins/openshift-sync/0.0.13/) which should fix this issue.

@bparees: Forgotten how to get the Jenkins image updated with this new version?

Comment 3 Ben Parees 2016-09-23 15:21:44 UTC
you need to ask Troy Dawson to package the plugin and once he's done that and mirrored it out, I need to kick off the jenkins image build.

Comment 4 Troy Dawson 2016-10-06 14:48:39 UTC
I have updated the plugin to jenkins-plugin-openshift-sync-0.0.13-1.el7 and pushed it to the mirrors.  I also think Ben has kicked off the image build, but I'm not positive.

Comment 5 Ben Parees 2016-10-06 21:46:42 UTC
Yes the centos image has been built.  Setting to modified awaiting a new rhel image.

Comment 6 Troy Dawson 2016-11-02 18:05:53 UTC
New jenkins-1 and jenkins-2 images have been built with the updated openshift-sync plugin.

Comment 8 Dongbo Yan 2016-11-03 09:20:18 UTC
Verified
brew-pulp.../openshift3/jenkins-1-rhel7  6f476676a4a1
openshift-sync	0.0.14

openshift v3.4.0.19+346a31d
kubernetes v1.4.0+776c994
etcd 3.1.0-rc.0

Cannot reproduce three issues in description.
But I find the number of jenkins jobs is less than pipeline build in openshift client, have reported a bug bz1386455 before. We can track in that bug, and close this one.

Comment 10 errata-xmlrpc 2017-01-18 12:52:04 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0066


Note You need to log in before you can comment on or make changes to this bug.