Red Hat Bugzilla – Bug 1389147
Jenkins pipeline build would trigger jenkins jobs endlessly with jenkins-1-centos7 image
Last modified: 2017-03-08 13:43 EST
Description of problem: Start a jenkins pipeline build, it would trigger jenkins jobs endlessly in jenkins server with openshift/jenkins-1-centos7. Version-Release number of selected component (if applicable): docker.io/openshift/jenkins-1-centos7 5c811ae94ada openshift-pipeline plugin 1.0.26 openshift v3.4.0.15+9c963ec kubernetes v1.4.0+776c994 etcd 3.1.0-alpha.1 How reproducible: Always Steps to Reproduce: 1.Login openshift, and create a project 2.Create pipeline bc and jenkins server $ oc new-app -f https://raw.githubusercontent.com/openshift/origin/master/examples/jenkins/pipeline/samplepipeline.json $ oc new-app jenkins-ephemeral 3.Start a pipeline build, check jenkins job in jenkins webconsole $ oc start-build sample-pipeline Actual results: trigger jenkins jobs endlessly in jenkins webconsole Expected results: only trigger one jenkins job in jenkins webconsole Additional info:
test with latest centos image docker.io/openshift/jenkins-1-centos7 5b7b34dd4518 openshift-login 0.3 openshift-pipeline 1.0.26 openshift-sync 0.0.14 openshift v3.4.0.16+cc70b72 kubernetes v1.4.0+776c994 etcd 3.1.0-rc.0 cannot reproduce this bug. the bug occur in the image openshift/jenkins-1-centos7 (5c811ae94ada) ,include plugin: openshift-login 0.2 openshift-pipeline 1.0.26 openshift-sync 0.0.13
So can we close this as fixed in upgraded version?
I think so, but I don't know what cause it
I'm not sure either but don't have the time to look into why I'm afraid. If it's not happening now then let's close this & reopen it if we see it again?
Sending back to QE to verify.. @Jimmi, are there any logs/tracing you'd like enabled if QE is able to recreate this?
I reproduce this issue with brew-pulp.../jenkins-1-rhel7 082a80fcb858 openshift-login 0.4 openshift-pipeline 1.0.28 openshift-sync 0.0.14 openshift v3.4.0.18+ada983f kubernetes v1.4.0+776c994 etcd 3.1.0-rc.0 Steps: 1.$oc new-app -f https://raw.githubusercontent.com/openshift/origin/master/examples/jenkins/pipeline/samplepipeline.json 2.$oc edit bc sample-pipeline -o json "jenkinsfile": "node('maven') {\nstage 'build'\nopenshiftBuild(buildConfig: 'ruby-example' modify jenkinsfile to use a non existent bc 3.$ oc start-build sample-pipeline actual result: pipeline build failed, will trigger jenkins jobs endlessly Additional info: If use an existing bc, jenkins build success, only trigger one jenkins job in jenkins webconsole Need I file a new bug for rhel image?
I can't reproduce that with the sync plugin against Jenkins. Receive an error in the build log that indicates the non-existent BC: `com.openshift.restclient.NotFoundException: buildconfigs "does not exist" not found buildconfigs "does not exist" not found`. I think we should wait for next release of sync plugin, update image & retest, but I'm pretty sure this is nothing to do with the sync plugin.
@Dongbo any idea where "does not exist" is coming from? is that the name of one of your buildconfigs? otherwise, discussed offline w/ Jimmi and we're going to update the sync plugin to v0.15 and build a new image and see if we can still recreate this there.
Also @Jimmi, where in the code are we catching the NotFoundException (is that within the sync plugin, or the openshift plugin? what path will the code take after catching that error?)
@jessica I think you indicated you hit this again today too? Maybe you can provide Jimmi w/ some additional logs/recreate steps.
I blew that environment away because it was hosing my system. What I do know is that I had instantiated the pipeline stuff yesterday, with openshift running in a vagrant VM. I suspended my laptop overnight and today I kicked another pipeline build and after that is when it started going crazy launching extra builds. I did notice eventually Jenkins crashed and tried to restart.
@bparees Catching the error is the build step's responsibility, not the sync plugin which does nothing but watch the status of the build, not handling stuff happening in the build pipeline.
@jimmi the sync plugin looks at buildconfigs (pipeline buildconfigs). it wasn't clear to me from the bug report or the log snippet you referenced whether this was the sync plugin having issues reading a pipeline buildconfig as part of the sync process, or the "Trigger a build" step having trouble reading a buildconfig as part of triggering a build. I take it you are stating that it is the latter?
I'm still unable to reproduce with the same pipeline referenced above. What I was saying was that the sync plugin doesn't handle anything happening in pipeline builds. It just starts, cancels & syncs back to OpenShift. It is the responsibility of the build step to properly handle errors such as BC not being found rather than the sync plugin.
yeah, I get that for the actual steps. Still isn't clear to me that the log error was coming from a "Step" as opposed to from the sync plugin synchronizing to a pipeline buildconfig object, though. That is, it's not clear which "BC" was not found: 1) an s2i/image buildconfig which is managed by the build step within the pipeline 2) a pipeline buildconfig which is managed by the sync plugin anyway let's get the new sync plugin and see if QE can still reproduce.
It's definitely coming from the step as the log is from com.openshift.restclient which isn't used in the sync plugin which uses the fabric8 openshift client so wouldn't log this error message: `com.openshift.restclient.NotFoundException: buildconfigs "does not exist" not found buildconfigs "does not exist" not found`.
got it, that was the detail I needed. thanks!
I've just run through the buildconfig above with input steps and can't reproduce, all looks smooth in the console too.
centos image has been updated w/ sync plugin v0.0.15, sending back to QE to have another look and see if they can provide specific recreate steps if the issue still occurs. @Jimmi any additional logging or other info you'd like QE to enable/collect if they still hit this?
To capture extra logs, go to `http://<jenkinsAddress>/log/levels` & add log levels for `io.fabric8` & `okhttp3` at level `all`. Try again & the logs should be populated.
Test with docker.io/openshift/jenkins-1-centos7@sha256:e913077282d9738c2bdd8e46b58cb04c79eb22eeffed511857a42d6d8904f35d openshift-login 0.8 openshift-pipeline 1.0.32 openshift-sync 0.0.15 openshift v3.4.0.23+24b1a58 kubernetes v1.4.0+776c994 etcd 3.1.0-rc.0 cannot reproduce this issue, no error in log Nov 10, 2016 8:39:18 AM INFO io.fabric8.jenkins.openshiftsync.BuildSyncRunListener upsertBuild Patching build in namespace dyan-centos with name: sample-pipeline-2 phase: Running Nov 10, 2016 8:39:19 AM INFO io.fabric8.jenkins.openshiftsync.BuildSyncRunListener upsertBuild Patching build in namespace dyan-centos with name: sample-pipeline-2 phase: Running Nov 10, 2016 8:39:20 AM INFO io.fabric8.jenkins.openshiftsync.BuildSyncRunListener upsertBuild Patching build in namespace dyan-centos with name: sample-pipeline-2 phase: Running