Bug 1149372

Summary: Jenkins won't schedule builds when jenkins is updated to latest LTS release
Product: OpenShift Online Reporter: Troy Dawson <tdawson>
Component: ImageAssignee: Ben Parees <bparees>
Status: CLOSED ERRATA QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.xCC: bparees, jokerman, mfojtik, mmccomas, vjuranek, xiuwang
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-10-10 13:32:05 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Troy Dawson 2014-10-03 22:19:04 UTC
Description of problem:
When jenkins is updated to 1.565.3 (the latest LTS release).  Everything appears to work correctly until jenkins attempts to schedule a build.

Version-Release number of selected component (if applicable):
jenkins-1.565.3-1.el6oso.noarch
jenkins-plugin-openshift-0.6.38-0.el6oso.x86_64
openshift-origin-cartridge-jenkins-1.25.1-1.el6oso.noarch
openshift-origin-cartridge-jenkins-client-1.25.1-1.el6oso.noarch


How reproducible:
100%

Steps to Reproduce:
rhc app create jenkins -a jenkins1
rhc app create testjenkins1 perl-5.10
rhc cartridge-add Jenkins --app testjenkins1
cd testjenkins1
vi index.pl
# Change "<title>Welcome to OpenShift</title>" to "<title>Welcome to JenkinsTest1</title>"
git commit -m "testing" -a
git push

Actual results:
$ git push
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 294 bytes, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Executing Jenkins build.
remote: 
remote: You can track your build at https://jenkins1-<domain>.rhcloud.com/job/testjenkins1-build
remote: 
remote: Waiting for build to schedule.......
remote: **BUILD FAILED/CANCELLED**
remote: Please see the Jenkins log for more details via 'rhc tail'
remote: !!!!!!!!
remote: Deployment Halted!
remote: If the build failed before the deploy step, your previous
remote: build is still running.  Otherwise, your application may be
remote: partially deployed or inaccessible.
remote: Fix the build and try again.
remote: !!!!!!!!
remote: An error occurred executing 'gear postreceive' (exit code: 1)
remote: Error message: CLIENT_ERROR: Failed to execute: 'control post-receive' for /var/lib/openshift/<UUID>/jenkins-client
remote: 
remote: For more details about the problem, try running the command again with the '--trace' option.
To ssh://<UUID>@testjenkins1-<domain>.rhcloud.com/~/git/testjenkins1.git/
   963ad48..0a029ca  master -> master


Expected results:
$ git push
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 296 bytes, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Executing Jenkins build.
remote: 
remote: You can track your build at https://jenkins1-<domain>.rhcloud.com/job/testjenkins1-build
remote: 
remote: Waiting for build to schedule..................................Done
remote: Waiting for job to complete..........................Done
remote: SUCCESS
remote: New build has been deployed.
remote: -------------------------
remote: Git Post-Receive Result: success
remote: Deployment completed with status: success
To ssh://<UUID>@testjenkins1-<domain>.rhcloud.com/~/git/testjenkins1.git/
   699fc75..caa2d1c  master -> master


Additional info:
Doing a rhc tail on the jenkins app give the following.

Oct 03, 2014 5:02:57 PM hudson.plugins.openshift.OpenShiftCloud provision
INFO: Provisioning new node for workload = 2 and label = null in domain tdawsontest
Oct 03, 2014 5:02:57 PM hudson.plugins.openshift.OpenShiftCloud provision
INFO: Cancelling build - Label is null
Oct 03, 2014 5:02:57 PM hudson.plugins.openshift.OpenShiftCloud cancelBuild
INFO: Cancelling build
Oct 03, 2014 5:02:57 PM hudson.plugins.openshift.OpenShiftCloud cancelItem
INFO: Cancelling Item 
Oct 03, 2014 5:02:57 PM hudson.plugins.openshift.OpenShiftCloud cancelItem
WARNING: Build null rawbldr has been canceled
Oct 03, 2014 5:02:57 PM hudson.triggers.SafeTimerTask run
SEVERE: Timer task hudson.slaves.NodeProvisioner$NodeProvisionerInvoker@209d8e failed
java.lang.UnsupportedOperationException: No Label
	at hudson.plugins.openshift.OpenShiftCloud.provision(OpenShiftCloud.java:402)
	at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:281)
	at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:51)
	at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:366)
	at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)

Comment 2 Michal Fojtik 2014-10-07 13:01:31 UTC
I discussed this today with Jenkins engineer and he said this might be a bug in Jenkins code, as the Label should never be null. He will try to reproduce it and fix.

Comment 4 XiuJuan Wang 2014-10-08 08:02:08 UTC
Verified with devenv_5219

Could launch jenkins job successfully with the latest LTS release.

1.rhc app create jenkins jenkins-1
2.rhc app create perl perl-5.10
3.rhc cartridge-add Jenkins-client-1 -a perl
4.Change something in local repo, and git push

Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 246 bytes, done.
Total 2 (delta 1), reused 0 (delta 0)
remote: Executing Jenkins build.
remote: 
remote: You can track your build at https://jenkins-test.dev.rhcloud.com/job/perltt-build
remote: 
remote: Waiting for build to schedule......Done
remote: Waiting for job to complete.....................Done
remote: SUCCESS
remote: New build has been deployed.
remote: -------------------------
remote: Git Post-Receive Result: success
remote: Deployment completed with status: success
To ssh://54351df589968e9c3b000107.rhcloud.com/~/git/perltt.git/
   d86e008..2dbc035  master -> master

Comment 5 Troy Dawson 2014-10-10 13:32:05 UTC
This fix has been pushed to production as errata.  Thanks to everyone for their work on fixing this.