Bug 1364949 – jenkinspipeline builds skip running status in openshift client after restart master api server

Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.

Bug 1364949 - jenkinspipeline builds skip running status in openshift client after restart master api server

Summary:	jenkinspipeline builds skip running status in openshift client after restart ...

Status:	CLOSED ERRATA

Aliases:	None

Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Image (Show other bugs)
Sub Component:
Version:	3.3.0
Hardware:	Unspecified Unspecified

Priority	medium Severity medium
Target Milestone:	---
Target Release:	---
Assigned To:	Jimmi Dyson
QA Contact:	Wang Haoran
Docs Contact:

URL:
Whiteboard:
Keywords:

Depends On:
Blocks:
	Show dependency tree / graph

Reported:	2016-08-08 05:31 EDT by XiuJuan Wang
Modified:	2017-03-08 13 EST (History)
CC List:	7 users (show)

See Also:
Fixed In Version:
Doc Type:	Bug Fix
Doc Text:	Cause: API server restart causes the Jenkins sync plugin to lose its connection to OpenShift Consequence: Pipeline builds are not properly executed in the Jenkins server Fix: Update sync plugin to handle connection loss when API server is restarted Result: Builds will be properly processed by the sync plugin and executed in Jenkins if the API server is restarted
Story Points:	---
Clone Of:
Environment:
Last Closed:	2017-01-18 07:52:08 EST
Type:	Bug
Regression:	---
Mount Type:	---
Documentation:	---
CRM:
Verified Versions:
Category:	---
oVirt Team:	---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

External Trackers
Tracker	ID	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2017:0066	normal	SHIPPED_LIVE	Red Hat OpenShift Container Platform 3.4 RPM Release Advisory	2017-01-18 12:23:26 EST

Groups: None (edit)

Description XiuJuan Wang 2016-08-08 05:31:58 EDT

Description of problem:
After restarting master api server, new triggered builds will become to complete/failed/cancelled directly from new status, no running staus in openshift client.

Works well before restart master.

Version-Release number of selected component (if applicable):
ci.dev.../openshift/jenkins-slave-nodejs-rhel7 965eb5e5766a
ci.dev.../openshift/jenkins-slave-maven-rhel7  7c6bbfb9656e
ci.dev.../openshift/jenkins-1-rhel7 bd3ab5d39d48

openshift v3.3.0.16
kubernetes v1.3.0+507d3a7

How reproducible:
always

Steps to Reproduce:
1.Create a jenkins pipeline app.
2.Trigger some builds from openshift client and jenkins webconsole
3.Restart master api server
4.Trigger some builds from openshift client and jenkins webconsole again.
5.Check builds from openshift client and jenkins webconsole

Actual results:
jenkinspipeline builds skip running status in openshift client

Expected results:
Should sync all build status between openshift client and jenkins webconsole.

Additional info:

Comment 1 Ben Parees 2016-08-16 09:35:24 EDT

also tracking as: https://github.com/fabric8io/openshift-jenkins-sync-plugin/issues/99

Comment 2 Jimmi Dyson 2016-09-21 10:14:49 EDT

This is a problem if the API server host is rebooted. If the API server process is stopped/started then it reconnects fine. This is a TCP timeout issue that affects oc client too (try it with `oc get pods -w` & then force reboot the API server - notice no timeout). I'm looking at implementing WS pings but not sure if API server supports pings (it should if it's spec compliant).

Comment 3 Jimmi Dyson 2016-09-21 11:56:27 EDT

This is a fix in kubernetes-client - PR https://github.com/fabric8io/kubernetes-client/pull/517. Will release client after merged & update jenkins plugin with new version after that.

Comment 4 Jimmi Dyson 2016-09-23 10:42:10 EDT

I've released openshift-jenkins-sync-plugin 0.0.13 (https://repo.jenkins-ci.org/releases/io/fabric8/jenkins/plugins/openshift-sync/0.0.13/) which should fix this issue.

@bparees: Forgotten how to get the Jenkins image updated with this new version?

Comment 5 Ben Parees 2016-10-06 17:47:44 EDT

centos image is available w/ the updated plugin.  rhel has not been rebuilt yet.

Comment 6 Troy Dawson 2016-11-02 14:05:14 EDT

New jenkins-1 and jenkins-2 images have been built with the updated openshift-sync plugin.

Comment 8 Dongbo Yan 2016-11-03 02:51:28 EDT

Verified
brew-pulp.../openshift3/jenkins-1-rhel7  6f476676a4a1
openshift-sync	0.0.14

openshift v3.4.0.19+346a31d
kubernetes v1.4.0+776c994
etcd 3.1.0-rc.0

Actual result:
After restart master api,new triggered pipeline builds could be running status, then change to complete.

Comment 10 errata-xmlrpc 2017-01-18 07:52:08 EST

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0066

Note You need to log in before you can comment on or make changes to this bug.