Bug 1176649

Summary: jenkins-client reports a failure during JBoss application build, but the Jenkins build logs show that the build was a success
Product: OpenShift Container Platform Reporter: Miheer Salunke <misalunk>
Component: ImageStreamsAssignee: Jason DeTiberus <jdetiber>
Status: CLOSED ERRATA QA Contact: libra bugs <libra-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 2.2.0CC: adellape, bleanhar, dmace, erich, gpei, jdetiber, jialiu, jokerman, libra-onpremise-devel, misalunk, mmccomas, pep, pruan
Target Milestone: ---Keywords: Reopened
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openshift-origin-cartridge-jenkins-client-1.25.1.1-1 Doc Type: Bug Fix
Doc Text:
When running the "git push" command for applications with a Jenkins Client cartridge, it was possible for developers to experience "502 Proxy Error" and "Deployment Halted" messages during the build process, even when the deployment actually succeeded. These messages were potentially confusing to developers trying to investigate the issue. This bug fix updates the Jenkins Client cartridge to use the Ruby Net::HTTP library to easily retry requests and correctly interpret responses. As a result, the Jenkins Client cartridge now gracefully handles errors and provides more informative messages. After applying this update, a cartridge upgrade is required.
Story Points: ---
Clone Of:
: 1176651 (view as bug list) Environment:
Last Closed: 2015-04-06 17:06:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1176651, 1196617    
Bug Blocks:    

Description Miheer Salunke 2014-12-22 17:47:32 UTC
Description of problem:

Deployment via git push fails if Jenkins is enabled on JBoss application but, the Jenkins build logs show that the build was a success

Version-Release number of selected component (if applicable):
2.2

How reproducible:
Always

Steps to Reproduce:
1. Follow steps as mentioned in https://developers.openshift.com/en/managing-continuous-integration.html
2. Deploy using git push 


Actual results:
Results after git push 

    Pushing to ssh://549410074382ec2a8b0000ad.com/~/git/jbosseap.git/
    Counting objects: 5, done.
    Delta compression using up to 4 threads.
    Compressing objects: 100% (3/3), done.
    Writing objects: 100% (3/3), 274 bytes, done.
    Total 3 (delta 2), reused 0 (delta 0)
    remote: Executing Jenkins build.
    remote:
    remote: You can track your build at https://jenkins-karandomain.rhcloud.com/job/jbosseap-build
    remote:
    remote: Waiting for build to schedule......Done
    remote: Waiting for job to complete.................................................../opt/rh/ruby193/root/usr/share/ruby/json/common.rb:155:in `parse': 757: unexpected token at '<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> (JSON::ParserError)
    remote: <html><head>
    remote: <title>502 Proxy Error</title>
    remote: </head><body>
    remote: <h1>Proxy Error</h1>
    remote: <p>The proxy server received an invalid
    remote: response from an upstream server.<br />
    remote: The proxy server could not handle the request <em><a href="/job/jbosseap-build/2/api/json">GET&nbsp;/job/jbosseap-build/2/api/json</a></em>.<p>
    remote: Reason: <strong>Error reading from remote server</strong></p></p>
    remote: <hr>
    remote: <address>Apache/2.2.15 (Red Hat) Server at jenkins-karandomain.rhcloud.com Port 443</address>
    remote: </body></html>'
    remote:         from /opt/rh/ruby193/root/usr/share/ruby/json/common.rb:155:in `parse'
    remote:         from /var/lib/openshift/549410074382ec2a8b0000ad/jenkins-client//bin/jenkins_build:35:in `get_job_info'
    remote:         from /var/lib/openshift/549410074382ec2a8b0000ad/jenkins-client//bin/jenkins_build:91:in `<main>'
    remote: !!!!!!!!
    remote: Deployment Halted!
    remote: If the build failed before the deploy step, your previous
    remote: build is still running.  Otherwise, your application may be
    remote: partially deployed or inaccessible.
    remote: Fix the build and try again.
    remote: !!!!!!!!
    remote: An error occurred executing 'gear postreceive' (exit code: 1)
    remote: Error message: CLIENT_ERROR: Failed to execute: 'control post-receive' for /var/lib/openshift/549410074382ec2a8b0000ad/jenkins-client
    remote:
    remote: For more details about the problem, try running the command again with the '--trace' option.
    To ssh://549410074382ec2a8b0000ad.com/~/git/jbosseap.git/
       2117087..b268c80  master -> master



Expected results:
The deployment using git push shall be a success.

Additional info:

Comment 3 Johnny Liu 2014-12-23 01:57:58 UTC
This should be duplicated with BZ#1125328, if not, pls reopen.

*** This bug has been marked as a duplicate of bug 1125328 ***

Comment 4 Josep 'Pep' Turro Mauri 2015-01-12 10:55:34 UTC
(In reply to Johnny Liu from comment #3)
> This should be duplicated with BZ#1125328, if not, pls reopen.

While related, it seems there are 2 bugs here:

  a) the jenkins-client side fails when it shouldn't (the build actually succeeds)

  b) the jenkins server generates ISEs

I'm reopening this bz to track issue (a) here, editing the subject a bit to clarify, while Bug 1125328 currently focuses on (b).

Comment 6 Josep 'Pep' Turro Mauri 2015-01-21 23:09:19 UTC
Comment #0 says this happens always (100% reproducible), which doesn't seem to be the case.

Sometimes the poll that jenkins-client does will get a 50x error from the server, when in fact the build is progressing correctly. The reasons behind that response from the server are investigated in bug 1125328.

When this happens, the jenkins_build script in jenkins-client will not detect it and try to parse the ISE output as normal JSON data (as if a 200 status code had been returned). This is because the script uses curl:

def get_jobs_info
  jobs = `curl -s --insecure https://${JENKINS_USERNAME}:${JENKINS_PASSWORD}@#{@hostname}#{@job_url}/api/json`
  result = $?

and curl's exit code will still be 0 (success) in this scenario.

One option could be to use wget instead: wget produces an error exit code if the server returns an error HTTP code. Something like:

  jobs = `wget -q -O- --no-check-certificate https://${JENKINS_USERNAME}:${JENKINS_PASSWORD}@#{@hostname}#{@job_url}/api/json`

but then this would result in jenkins_build exiting with an error code too - not sure that's what we want.

Eric suggested doing a few retries in this scenario. Maybe a combination of wget instead of curl + retries would make the client more robust.

Comment 18 Gaoyun Pei 2015-03-18 07:27:45 UTC
Verify this bug with openshift-origin-cartridge-jenkins-client-1.25.1.1-1.el6op.noarch

The client side report success when triggering jenkins build job on vhost plugin enabled node.

Create jbosseap app with jenkins-client and trigger jenkins build.

[root@broker app1]# git push
Counting objects: 3, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 254 bytes, done.
Total 2 (delta 1), reused 0 (delta 0)
remote: Executing Jenkins build.
remote: 
remote: You can track your build at https://jj-yes.ose22-manual.com.cn/job/app1-build
remote: 
remote: Waiting for build to schedule.......................................................Done
remote: Waiting for job to complete.....................................................................................................................Done
remote: SUCCESS
remote: New build has been deployed.
remote: -------------------------
remote: Git Post-Receive Result: success
remote: Deployment completed with status: success
To ssh://yes-app1-1.com.cn/~/git/app1.git/
   55853b1..4b4fb35  master -> master

Tried jenkins build 3 times, got 502 error once during building in /var/log/httpd/openshift_log, the client side still could finish job successfully, so move this bug to verified.

Comment 22 errata-xmlrpc 2015-04-06 17:06:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2015-0779.html