Bug 1369740 - Pipeline build should be failed when related build fail
Summary: Pipeline build should be failed when related build fail
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: ImageStreams
Version: 3.3.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: ---
Assignee: Gabe Montero
QA Contact: Wang Haoran
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-24 09:34 UTC by Dongbo Yan
Modified: 2017-03-08 18:26 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: the build step in OpenShift Pipeline Plugin for Jenkins did not fully check during a certain wait loop the status of Builds running in OpenShift Consequence: Jobs in Jenkins with OpenShift Pipeline Plugin builds steps would wait the full timeout duration unnecessarily before failing a given Jenkins job if the corresponding Build in OpenShift failed much sooner. Fix: Properly check the OpenShift Build status during the wait polling from Jenkins job leveraging the OpenShift Pipeline Plugin for Jenkins build step. Result: Jenkins jobs will fail immediately if a corresponding Build in OpenShift fails.
Clone Of:
Environment:
Last Closed: 2016-09-27 09:10:49 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2016:1937 0 normal SHIPPED_LIVE Red Hat OpenShift Enterprise Jenkins image bug fix and enhancement update 2016-09-27 13:05:12 UTC

Description Dongbo Yan 2016-08-24 09:34:15 UTC
Description of problem:
Pipeline build will not fail when related build "ruby-sample-build-1" fail

Version-Release number of selected component (if applicable):
brew-pulp...jenkins-1-rhel7 (7e4706a0e228)
openshift v3.3.0.24-dirty
kubernetes v1.3.0+507d3a7
etcd 2.3.0+git

How reproducible:
Always

Steps to Reproduce:
1.Create pipeline bc and other resources
 $ oc new-app -f https://raw.githubusercontent.com/openshift/origin/master/examples/jenkins/pipeline/samplepipeline.json
 $ oc new-app jenkins-ephemeral
2.Edit bc ruby-sample-build to use an incorrect source code
3.Start new pipeline build
 $ oc start-build sample-pipeline
4.Wait build "ruby-sample-build-1" failed, check pipeline build

Actual results:
Pipeline build will not fail, keep running

Expected results:
Pipeline build and "ruby-sample-build-1" build both fail

Additional info:
If cancel build "ruby-sample-build-1", pipeline build will fail

Comment 1 Gabe Montero 2016-08-24 17:55:47 UTC
OK, I've "repro'ed" and know what is going on.  It technically will fail, you just have to wait long enough.  The default build wait time is 15 minutes.

That said, there was a change on Jul 25 https://github.com/openshift/jenkins-plugin/commit/a571787293a757c4cf2a127bfccddff1d4b9c49f that actually did not make v1.0.21, though that might seem confusing if you look at the commits on github.com/openshift/jenkins-plugin.

v1.0.21 cut on jenkinsci/openshift-pipeline-plugin on Jul 13, but I didn't merge the version change to openshift/jenkins-pipeline until Jul 25 as well.

I'll confirm the fix noted above performs the fast fail.  If so, I would think we only need one of the recent bug's opened to drive the plugin version change / image gen.

If for some reason the existing fix does not fail fast in the pipeline / dsl scenario (the work I did upstream with Oleg Popov only verified fast fail for classic freestyle jobs), I'll use this bug to drive the additional change to the plugin.

Comment 2 Gabe Montero 2016-08-24 18:15:13 UTC
Yep, the fast fail fix handled this:

OpenShift Build pipelineproject/sample-pipeline-2
[Pipeline] node
Running on maven-e568b88dcdb in /tmp/workspace/sample-pipeline
[Pipeline] {
[Pipeline] stage (build)
Entering stage build
Proceeding
[Pipeline] openshiftBuild


Starting the "Trigger OpenShift Build" step with build config "ruby-sample-build" from the project "pipelineproject".
  Started build "ruby-sample-build-2" and waiting for build completion ...
Downloading "https://github.com/openshift/foo-bar" ...
error: build error: failed to fetch requested repository "https://github.com/openshift/foo-bar" with provided credentials
Downloading "https://github.com/openshift/foo-bar" ...
error: build error: failed to fetch requested repository "https://github.com/openshift/foo-bar" with provided credentials
Downloading "https://github.com/openshift/foo-bar" ...
error: build error: failed to fetch requested repository "https://github.com/openshift/foo-bar" with provided credentials


Exiting "Trigger OpenShift Build" unsuccessfully; build "ruby-sample-build-2" did not complete successfully within the configured timeout of "900000" ms; last reported status:  [Failed].
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: "Trigger OpenShift Build" failed
Finished: FAILURE

After talking with Ben, I'll leave this open (since it is a separate commit in the repro for this fix) and move onto QA along with the other bugs when the image gets generated.

Comment 3 Gabe Montero 2016-08-25 23:43:43 UTC
brew pulp latest has v1.0.22 of the plugin

QE can attempt to verify

Comment 4 Dongbo Yan 2016-08-26 02:54:54 UTC
Verified 
brew-pulp...openshift3/jenkins-1-rhel7   latest   023723e2e0ff    5 hours ago

Pipeline build fail once ruby-sample-build-1 build failed

Starting the "Trigger OpenShift Build" step with build config "ruby-sample-build" from the project "pipelineproject".
  Started build "ruby-sample-build-1" and waiting for build completion ...
Cloning "https://github.com/openshiftii/ruby-hello-world.git" ...
error: build error: failed to fetch requested repository "https://github.com/openshiftii/ruby-hello-world.git" with provided credentials
Cloning "https://github.com/openshiftii/ruby-hello-world.git" ...
error: build error: failed to fetch requested repository "https://github.com/openshiftii/ruby-hello-world.git" with provided credentials
Cloning "https://github.com/openshiftii/ruby-hello-world.git" ...
error: build error: failed to fetch requested repository "https://github.com/openshiftii/ruby-hello-world.git" with provided credentials


Exiting "Trigger OpenShift Build" unsuccessfully; build "ruby-sample-build-1" did not complete successfully within the configured timeout of "900000" ms; last reported status:  [Failed].
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: "Trigger OpenShift Build" failed
Finished: FAILURE

Comment 6 errata-xmlrpc 2016-09-27 09:10:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2016:1937


Note You need to log in before you can comment on or make changes to this bug.