Bug 1411336

Summary: [dev-preview-int] Pipeline builds stuck in "New" state
Product: OpenShift Online Reporter: Steve Speicher <sspeiche>
Component: BuildAssignee: Jimmi Dyson <jdyson>
Status: CLOSED DUPLICATE QA Contact: Wang Haoran <haowang>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 3.xCC: aos-bugs, bparees, cewong, yasun
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-01-10 18:19:28 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Steve Speicher 2017-01-09 14:11:02 UTC
Description of problem:
In running the base pipeline example, I set it up to run periodically (every 30 minutes). Over the last 6 hours it hasn't been able to successfully start a new pipeline build.

Version-Release number of selected component (if applicable):
OpenShift Master:
v3.4.0.38 (online version 3.4.0.11)
Kubernetes Master:
v1.4.0+776c994
Jenkins ver. 2.19.3
Jenkins plugins-
  OpenShift Pipeline Jenkins Plugin: 1.0.37
  OpenShift Sync: 0.1.5

How reproducible:
Deploy sample pipeline and repeatedly run the pipeline, observe failures.


Steps to Reproduce:
1. Login to dev-preview-int
2. Create project
3. Deploy simple sample 
    oc new-app -f https://raw.githubusercontent.com/openshift/origin/master/examples/jenkins/pipeline/samplepipeline.yaml
4. Go to Jenkins console and setup periodic builds

Actual results:
Pipelines eventually stuck in "New" state in OpenShift and no corresponding build jobs in Jenkins, with many log entries in Jenkins like:

Jan 09, 2017 1:30:10 PM io.fabric8.jenkins.openshiftsync.BuildConfigWatcher onInitialBuildConfigs
SEVERE: Failed to update job
java.lang.IllegalArgumentException: Jenkins already contains an item 'sspeiche-pipeline-sample-pipeline'


Expected results:
My pipeline builds successfully

Additional info:

Project: sspeiche-pipeline

$oc describe build sample-pipeline-12
Name:		sample-pipeline-12
Namespace:	sspeiche-pipeline
Created:	5 hours ago
Labels:		app=jenkins-pipeline-example
		buildconfig=sample-pipeline
		name=sample-pipeline
		openshift.io/build-config.name=sample-pipeline
		openshift.io/build.start-policy=Serial
		template=application-template-sample-pipeline
Annotations:	openshift.io/build-config.name=sample-pipeline
		openshift.io/build.number=12

Status:		New
Duration:	waiting for 5h31m42s
Build Config:	sample-pipeline
Build Pod:	sample-pipeline-12-build

Strategy:	JenkinsPipeline
Jenkinsfile contents:
  node('nodejs') {
    stage('build') {
      openshiftBuild(buildConfig: 'nodejs-mongodb-example', showBuildLogs: 'true')
    }
    stage('deploy') {
      openshiftDeploy(deploymentConfig: 'nodejs-mongodb-example')
    }
  }
Empty Source:	no input source provided

No events.

=====================================
Jenkins log information (many entries)

Jan 09, 2017 1:30:00 PM io.fabric8.jenkins.openshiftsync.BuildConfigWatcher eventReceived
WARNING: Caught: java.lang.IllegalArgumentException: Jenkins already contains an item 'sspeiche-pipeline-sample-pipeline'
java.lang.IllegalArgumentException: Jenkins already contains an item 'sspeiche-pipeline-sample-pipeline'
	at hudson.model.ItemGroupMixIn.createProjectFromXML(ItemGroupMixIn.java:264)
	at jenkins.model.Jenkins.createProjectFromXML(Jenkins.java:3689)
	at io.fabric8.jenkins.openshiftsync.BuildConfigWatcher$2.call(BuildConfigWatcher.java:214)
	at io.fabric8.jenkins.openshiftsync.BuildConfigWatcher$2.call(BuildConfigWatcher.java:159)
	at hudson.security.ACL.impersonate(ACL.java:221)
	at io.fabric8.jenkins.openshiftsync.BuildConfigWatcher.upsertJob(BuildConfigWatcher.java:159)
	at io.fabric8.jenkins.openshiftsync.BuildConfigWatcher.modifyJob(BuildConfigWatcher.java:238)
	at io.fabric8.jenkins.openshiftsync.BuildConfigWatcher.eventReceived(BuildConfigWatcher.java:149)
	at io.fabric8.jenkins.openshiftsync.BuildConfigWatcher.eventReceived(BuildConfigWatcher.java:65)
	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager$1.onMessage(WatchConnectionManager.java:215)
	at okhttp3.internal.ws.RealWebSocket$1.onMessage(RealWebSocket.java:62)
	at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:242)
	at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:108)
	at okhttp3.internal.ws.RealWebSocket.readMessage(RealWebSocket.java:97)
	at okhttp3.ws.WebSocketCall.createWebSocket(WebSocketCall.java:152)
	at okhttp3.ws.WebSocketCall.access$000(WebSocketCall.java:41)
	at okhttp3.ws.WebSocketCall$1.onResponse(WebSocketCall.java:97)
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:126)
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Jan 09, 2017 1:30:10 PM io.fabric8.jenkins.openshiftsync.BuildConfigWatcher onInitialBuildConfigs
SEVERE: Failed to update job
java.lang.IllegalArgumentException: Jenkins already contains an item 'sspeiche-pipeline-sample-pipeline'
	at hudson.model.ItemGroupMixIn.createProjectFromXML(ItemGroupMixIn.java:264)
	at jenkins.model.Jenkins.createProjectFromXML(Jenkins.java:3689)
	at io.fabric8.jenkins.openshiftsync.BuildConfigWatcher$2.call(BuildConfigWatcher.java:214)
	at io.fabric8.jenkins.openshiftsync.BuildConfigWatcher$2.call(BuildConfigWatcher.java:159)
	at hudson.security.ACL.impersonate(ACL.java:221)
	at io.fabric8.jenkins.openshiftsync.BuildConfigWatcher.upsertJob(BuildConfigWatcher.java:159)
	at io.fabric8.jenkins.openshiftsync.BuildConfigWatcher.onInitialBuildConfigs(BuildConfigWatcher.java:129)
	at io.fabric8.jenkins.openshiftsync.BuildConfigWatcher.access$200(BuildConfigWatcher.java:65)
	at io.fabric8.jenkins.openshiftsync.BuildConfigWatcher$1.doRun(BuildConfigWatcher.java:88)
	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:50)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Comment 1 Steve Speicher 2017-01-10 03:17:53 UTC
I can make it go a little further by restarting the sync plugin, by scaling Jenkins to 0 and then back up to 1. Am I the only one experiencing this level of instability?

Comment 2 yasun 2017-01-10 07:31:26 UTC
I can not reproduce it on dev-preview-int.  Followed the above steps, and set a "H/10 * * * *" schedule build. Till now, there have been 11 success builds:
[root@ymsun online-test]# oc get builds sample-pipeline-11 nodejs-mongodb-example-11
NAME                        TYPE              FROM          STATUS     STARTED          DURATION
sample-pipeline-11          JenkinsPipeline                 Complete   11 minutes ago   1m13s
nodejs-mongodb-example-11   Source            Git@343689f   Complete   11 minutes ago   33s

Comment 3 Steve Speicher 2017-01-10 14:32:11 UTC
I consistently hit this still, perhaps 1 in 10 is when I see it.

$ oc get builds                        
NAME                        TYPE              FROM          STATUS     STARTED             DURATION
nodejs-mongodb-example-31   Source            Git@343689f   Failed     5 hours ago         3m52s
nodejs-mongodb-example-32   Source            Git@343689f   Complete   5 hours ago         39s
nodejs-mongodb-example-33   Source            Git@343689f   Complete   4 hours ago         43s
nodejs-mongodb-example-34   Source            Git@343689f   Complete   3 hours ago         47s
nodejs-mongodb-example-35   Source            Git@343689f   Complete   2 hours ago         1m5s
sample-pipeline-31          JenkinsPipeline                 Failed     6 hours ago         5m35s
sample-pipeline-32          JenkinsPipeline                 Complete   5 hours ago         1m59s
sample-pipeline-33          JenkinsPipeline                 Complete   4 hours ago         2m1s
sample-pipeline-34          JenkinsPipeline                 Complete   3 hours ago         1m55s
sample-pipeline-35          JenkinsPipeline                 Complete   2 hours ago         3m5s
sample-pipeline-36          JenkinsPipeline                 Running    About an hour ago   1h1m33s
sample-pipeline-37          JenkinsPipeline                 New                            


$oc describe builds sample-pipeline-36
Name:		sample-pipeline-36
Namespace:	sspeiche-pipeline
Created:	58 minutes ago
Labels:		app=jenkins-pipeline-example
		buildconfig=sample-pipeline
		name=sample-pipeline
		openshift.io/build-config.name=sample-pipeline
		openshift.io/build.start-policy=Serial
		template=application-template-sample-pipeline
Annotations:	openshift.io/build-config.name=sample-pipeline
		openshift.io/build.number=36
		openshift.io/jenkins-build-uri=https://jenkins-sspeiche-pipeline.1ec1.dev-preview-int.openshiftapps.com/job/sspeiche-pipeline-sample-pipeline/36/
		openshift.io/jenkins-log-url=https://jenkins-sspeiche-pipeline.1ec1.dev-preview-int.openshiftapps.com/job/sspeiche-pipeline-sample-pipeline/36/consoleText
		openshift.io/jenkins-status-json={"_links":{"self":{"href":"https://jenkins-sspeiche-pipeline.1ec1.dev-preview-int.openshiftapps.com/job/sspeiche-pipeline-sample-pipeline/36/wfapi/describe"}},"id":"36","name":"#36","status":"IN_PROGRESS","startTimeMillis":1484055000943,"endTimeMillis":1484058529349,"durationMillis":0,"queueDurationMillis":3528406,"pauseDurationMillis":0,"stages":[]}

Status:		Running
Started:	Tue, 10 Jan 2017 08:30:00 EST
Duration:	running for 58m50s
Build Config:	sample-pipeline
Build Pod:	sample-pipeline-36-build

Strategy:	JenkinsPipeline
Jenkinsfile contents:
  node('nodejs') {
    stage('build') {
      openshiftBuild(buildConfig: 'nodejs-mongodb-example', showBuildLogs: 'true')
    }
    stage('deploy') {
      openshiftDeploy(deploymentConfig: 'nodejs-mongodb-example')
    }
  }
Empty Source:	no input source provided

No events.

Comment 4 Ben Parees 2017-01-10 18:19:28 UTC

*** This bug has been marked as a duplicate of bug 1406306 ***