Bug 1176649 - jenkins-client reports a failure during JBoss application build, but the Jenkins build logs show that the build was a success
Summary: jenkins-client reports a failure during JBoss application build, but the Jenk...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: ImageStreams
Version: 2.2.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: ---
Assignee: Jason DeTiberus
QA Contact: libra bugs
URL:
Whiteboard:
Depends On: 1176651 1196617
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-12-22 17:47 UTC by Miheer Salunke
Modified: 2019-07-11 08:29 UTC (History)
13 users (show)

Fixed In Version: openshift-origin-cartridge-jenkins-client-1.25.1.1-1
Doc Type: Bug Fix
Doc Text:
When running the "git push" command for applications with a Jenkins Client cartridge, it was possible for developers to experience "502 Proxy Error" and "Deployment Halted" messages during the build process, even when the deployment actually succeeded. These messages were potentially confusing to developers trying to investigate the issue. This bug fix updates the Jenkins Client cartridge to use the Ruby Net::HTTP library to easily retry requests and correctly interpret responses. As a result, the Jenkins Client cartridge now gracefully handles errors and provides more informative messages. After applying this update, a cartridge upgrade is required.
Clone Of:
: 1176651 (view as bug list)
Environment:
Last Closed: 2015-04-06 17:06:03 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2015:0779 0 normal SHIPPED_LIVE Red Hat OpenShift Enterprise 2.2.5 bug fix and enhancement update 2015-04-06 21:05:45 UTC

Internal Links: 1134206

Description Miheer Salunke 2014-12-22 17:47:32 UTC
Description of problem:

Deployment via git push fails if Jenkins is enabled on JBoss application but, the Jenkins build logs show that the build was a success

Version-Release number of selected component (if applicable):
2.2

How reproducible:
Always

Steps to Reproduce:
1. Follow steps as mentioned in https://developers.openshift.com/en/managing-continuous-integration.html
2. Deploy using git push 


Actual results:
Results after git push 

    Pushing to ssh://549410074382ec2a8b0000ad.com/~/git/jbosseap.git/
    Counting objects: 5, done.
    Delta compression using up to 4 threads.
    Compressing objects: 100% (3/3), done.
    Writing objects: 100% (3/3), 274 bytes, done.
    Total 3 (delta 2), reused 0 (delta 0)
    remote: Executing Jenkins build.
    remote:
    remote: You can track your build at https://jenkins-karandomain.rhcloud.com/job/jbosseap-build
    remote:
    remote: Waiting for build to schedule......Done
    remote: Waiting for job to complete.................................................../opt/rh/ruby193/root/usr/share/ruby/json/common.rb:155:in `parse': 757: unexpected token at '<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> (JSON::ParserError)
    remote: <html><head>
    remote: <title>502 Proxy Error</title>
    remote: </head><body>
    remote: <h1>Proxy Error</h1>
    remote: <p>The proxy server received an invalid
    remote: response from an upstream server.<br />
    remote: The proxy server could not handle the request <em><a href="/job/jbosseap-build/2/api/json">GET&nbsp;/job/jbosseap-build/2/api/json</a></em>.<p>
    remote: Reason: <strong>Error reading from remote server</strong></p></p>
    remote: <hr>
    remote: <address>Apache/2.2.15 (Red Hat) Server at jenkins-karandomain.rhcloud.com Port 443</address>
    remote: </body></html>'
    remote:         from /opt/rh/ruby193/root/usr/share/ruby/json/common.rb:155:in `parse'
    remote:         from /var/lib/openshift/549410074382ec2a8b0000ad/jenkins-client//bin/jenkins_build:35:in `get_job_info'
    remote:         from /var/lib/openshift/549410074382ec2a8b0000ad/jenkins-client//bin/jenkins_build:91:in `<main>'
    remote: !!!!!!!!
    remote: Deployment Halted!
    remote: If the build failed before the deploy step, your previous
    remote: build is still running.  Otherwise, your application may be
    remote: partially deployed or inaccessible.
    remote: Fix the build and try again.
    remote: !!!!!!!!
    remote: An error occurred executing 'gear postreceive' (exit code: 1)
    remote: Error message: CLIENT_ERROR: Failed to execute: 'control post-receive' for /var/lib/openshift/549410074382ec2a8b0000ad/jenkins-client
    remote:
    remote: For more details about the problem, try running the command again with the '--trace' option.
    To ssh://549410074382ec2a8b0000ad.com/~/git/jbosseap.git/
       2117087..b268c80  master -> master



Expected results:
The deployment using git push shall be a success.

Additional info:

Comment 3 Johnny Liu 2014-12-23 01:57:58 UTC
This should be duplicated with BZ#1125328, if not, pls reopen.

*** This bug has been marked as a duplicate of bug 1125328 ***

Comment 4 Josep 'Pep' Turro Mauri 2015-01-12 10:55:34 UTC
(In reply to Johnny Liu from comment #3)
> This should be duplicated with BZ#1125328, if not, pls reopen.

While related, it seems there are 2 bugs here:

  a) the jenkins-client side fails when it shouldn't (the build actually succeeds)

  b) the jenkins server generates ISEs

I'm reopening this bz to track issue (a) here, editing the subject a bit to clarify, while Bug 1125328 currently focuses on (b).

Comment 6 Josep 'Pep' Turro Mauri 2015-01-21 23:09:19 UTC
Comment #0 says this happens always (100% reproducible), which doesn't seem to be the case.

Sometimes the poll that jenkins-client does will get a 50x error from the server, when in fact the build is progressing correctly. The reasons behind that response from the server are investigated in bug 1125328.

When this happens, the jenkins_build script in jenkins-client will not detect it and try to parse the ISE output as normal JSON data (as if a 200 status code had been returned). This is because the script uses curl:

def get_jobs_info
  jobs = `curl -s --insecure https://${JENKINS_USERNAME}:${JENKINS_PASSWORD}@#{@hostname}#{@job_url}/api/json`
  result = $?

and curl's exit code will still be 0 (success) in this scenario.

One option could be to use wget instead: wget produces an error exit code if the server returns an error HTTP code. Something like:

  jobs = `wget -q -O- --no-check-certificate https://${JENKINS_USERNAME}:${JENKINS_PASSWORD}@#{@hostname}#{@job_url}/api/json`

but then this would result in jenkins_build exiting with an error code too - not sure that's what we want.

Eric suggested doing a few retries in this scenario. Maybe a combination of wget instead of curl + retries would make the client more robust.

Comment 18 Gaoyun Pei 2015-03-18 07:27:45 UTC
Verify this bug with openshift-origin-cartridge-jenkins-client-1.25.1.1-1.el6op.noarch

The client side report success when triggering jenkins build job on vhost plugin enabled node.

Create jbosseap app with jenkins-client and trigger jenkins build.

[root@broker app1]# git push
Counting objects: 3, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 254 bytes, done.
Total 2 (delta 1), reused 0 (delta 0)
remote: Executing Jenkins build.
remote: 
remote: You can track your build at https://jj-yes.ose22-manual.com.cn/job/app1-build
remote: 
remote: Waiting for build to schedule.......................................................Done
remote: Waiting for job to complete.....................................................................................................................Done
remote: SUCCESS
remote: New build has been deployed.
remote: -------------------------
remote: Git Post-Receive Result: success
remote: Deployment completed with status: success
To ssh://yes-app1-1.com.cn/~/git/app1.git/
   55853b1..4b4fb35  master -> master

Tried jenkins build 3 times, got 502 error once during building in /var/log/httpd/openshift_log, the client side still could finish job successfully, so move this bug to verified.

Comment 22 errata-xmlrpc 2015-04-06 17:06:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0779.html


Note You need to log in before you can comment on or make changes to this bug.