Bug 962324 - Jenkins build fail after migration and recreating jenkins-client
Jenkins build fail after migration and recreating jenkins-client
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Containers (Show other bugs)
1.x
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Dan Mace
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-05-13 04:14 EDT by Wei Sun
Modified: 2013-06-11 00:03 EDT (History)
6 users (show)

See Also:
Fixed In Version: devenv_3241+
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-06-11 00:03:10 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Wei Sun 2013-05-13 04:14:33 EDT
Description of problem:
After migration,can not Jenkins build the app successfully.It shows some error messages,like:
 remote: Failed to execute: 'control post-receive' for /var/lib/openshift/5190824acc2e782dd8000003/jenkins-client: rc(1)


Version-Release number of selected component (if applicable):
Upgrading devenv-stage_351 to devenv_3219 and migrate

How reproducible:
Always

Steps to Reproduce:
1. Launch a devenv-stage_351 ami
2. Create an app with jenkins-client-1.4 (v1 format)
3. scp -r ~/devenv-local and /etc/yum.repos.d/local.repo from devenv_3219 instance to same location on devenv-stage_351 instance
4. yum -y update --enablerepo devenv-local (the instance will be default to v2 format after upgrade)
5. rhc-v2-migrate --version 2.0.28
6. Git clone the app and Jenkins build
  
Actual results:
[sunwei@dhcp-8-229 phpjk]$ gitpush 
[master 0fb450f] test
 1 file changed, 1 insertion(+)
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 277 bytes, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Executing Jenkins build.
remote: 
remote: You can track your build at https://jk-wsunmigrate3129.dev.rhcloud.com/job/phpjk-build
remote: 
remote: Waiting for build to schedule......................................Done
remote: Waiting for job to complete................................................Done
remote: FAILED
remote: !!!!!!!!
remote: Deployment Halted!
remote: If the build failed before the deploy step, your previous
remote: build is still running.  Otherwise, your application may be
remote: partially deployed or inaccessible.
remote: Fix the build and try again.
remote: !!!!!!!!
remote: Failed to execute: 'control post-receive' for /var/lib/openshift/5190824acc2e782dd8000003/jenkins-client: rc(1)
remote: stdout: #<IO:0x000000011ef298>
remote: stderr#<IO:0x000000011ef220>
remote: /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/v2_cart_model.rb:950:in `block in do_control_with_directory'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/v2_cart_model.rb:776:in `process_cartridges'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/v2_cart_model.rb:923:in `do_control_with_directory'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/v2_cart_model.rb:799:in `do_control'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/application_container.rb:416:in `post_receive'
remote: /usr/bin/gear:133:in `block (3 levels) in <main>'
remote: /usr/bin/gear:80:in `do_command'
remote: /usr/bin/gear:132:in `block (2 levels) in <main>'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:155:in `run'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:385:in `run_active_command'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:62:in `run!'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/delegates.rb:11:in `run!'
remote: /opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/import.rb:10:in `block in <top (required)>'
To ssh://5190824acc2e782dd8000003@phpjk-wsunmigrate3129.dev.rhcloud.com/~/git/phpjk.git/
   c4964a1..0fb450f  master -> master


Expected results:
Can Jenkins build successfully.

Additional info:
Comment 1 Jhon Honce 2013-05-13 11:47:51 EDT
Message to users on Jenkins migration:

To support future features, during the new cartridge refactor we took the opportunity to make enhancements to our Jenkins support. Unfortunately, these changes require some work on your part.
 
Steps to update Jenkins jobs to work with new cartridge format:

1. Rename your existing Jenkins jobs (This backs up any changes you have made)

2. Remove your existing Jenkins-client cartridges from your applications

3. Re-add the Jenkins-client to your applications (This will create a new job with all it's new goodness)

Step#4 is only required if you have modified the default Jenkins job.

4. Edit the new Jenkins job and enter any custom logic from your renamed old Jenkins jobs

Sorry for the inconvenience, as we continue to strive to bring you the best PaaS we can.
Comment 2 chunchen 2013-05-14 03:08:32 EDT
this issue is reproduced on devenv_3223, please refer to the following results:

Steps to Reproduce:
1. Launch a devenv-stage_351 ami
2. Create an app with jenkins-client-1.4 (v1 format)
3. scp -r ~/devenv-local and /etc/yum.repos.d/local.repo from devenv_3219 instance to same location on devenv-stage_351 instance
4. yum -y update --enablerepo devenv-local (the instance will be default to v2 format after upgrade)
5. rhc-v2-migrate --version 2.0.28
6. Rename existing Jenkins jobs
access <jenkins-server-url>/job/$appname-build/configure, rename the value of "Project name"
7. Remove Jenkins-client cartridge from this app
8. Re-add Jenkins-client cartridge to this app
9. Make some changes and Jenkins build

After Step9,met the same error messages as before.
Comment 3 chunchen 2013-05-14 03:40:28 EDT
For the jenkins error logs as below:

<-------------snip-------------->
+ /usr/libexec/openshift/cartridges/v2/jenkins/bin/git_ssh_wrapper.sh a841b0c2bc3e11e29a6b22000a880b50@cews1jkns-cdm.dev.rhcloud.com 'gear stop --conditional'
Stopping gear...
cat: /var/lib/openshift/a841b0c2bc3e11e29a6b22000a880b50/jbossews//run/jboss.pid: No such file or directory
Failed to execute: 'control stop' for /var/lib/openshift/a841b0c2bc3e11e29a6b22000a880b50/jbossews: rc(1)
stdout: #<IO:0x00000001117398>
stderr#<IO:0x00000001117320>
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/v2_cart_model.rb:950:in `block in do_control_with_directory'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/v2_cart_model.rb:776:in `process_cartridges'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/v2_cart_model.rb:923:in `do_control_with_directory'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/v2_cart_model.rb:799:in `do_control'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/v2_cart_model.rb:1156:in `stop_cartridge'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/v2_cart_model.rb:1023:in `block in stop_gear'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/v2_cart_model.rb:78:in `block in each_cartridge'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/v2_cart_model.rb:783:in `block in process_cartridges'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/v2_cart_model.rb:780:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/v2_cart_model.rb:780:in `process_cartridges'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/v2_cart_model.rb:76:in `each_cartridge'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/v2_cart_model.rb:1022:in `stop_gear'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/application_container.rb:324:in `stop_gear'
/usr/bin/gear:210:in `block (3 levels) in <main>'
/usr/bin/gear:80:in `do_command'
/usr/bin/gear:202:in `block (2 levels) in <main>'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:155:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:385:in `run_active_command'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:62:in `run!'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/delegates.rb:11:in `run!'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/import.rb:10:in `block in <top (required)>'
Build step 'Execute shell' marked build as failure
Archiving artifacts
Finished: FAILURE
Comment 4 Dan McPherson 2013-05-14 10:00:08 EDT
Dan,

  This looks like ews doesn't handle the pid file not being in place.

cat: /var/lib/openshift/a841b0c2bc3e11e29a6b22000a880b50/jbossews//run/jboss.pid: No such file or directory
Failed to execute: 'control stop' for /var/lib/openshift/a841b0c2bc3e11e29a6b22000a880b50/jbossews: rc(1)


It may need to be fixed during the migration but should probably also be handled by the control script in general.
Comment 5 Dan Mace 2013-05-16 11:01:28 EDT
Should be resolved by https://github.com/openshift/origin-server/pull/2507 and https://github.com/openshift/li/pull/1390 (although the stack traces are another issue covered by bug 960809).
Comment 6 openshift-github-bot 2013-05-16 12:30:00 EDT
Commit pushed to master at https://github.com/openshift/li

https://github.com/openshift/li/commit/da7e991e703d24d3a7f676d435665be9dbba9049
Bug 962324: Fix jbossews stop logic to tolerate missing pidfile
Comment 7 openshift-github-bot 2013-05-16 12:30:04 EDT
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/26f7bed6eef569302eefd578d301a0c502310df5
Bug 962324: Fix jbossews stop logic to tolerate missing pidfile
Comment 8 Rony Gong 2013-05-17 04:46:58 EDT
Retest on migration to devenv_3235, Failed
Steps:
1. create diy, jbossews-2.0 app and embed jenkins-client
2. do migration
3. do change to this application, then git push, jenkins build trigged, Failed with jenkins log:

Started by user Jenkins System Builder
Building remotely on q2jew2bldr in workspace app-root/data/jenkins/workspace/q2jew2-build
Checkout:q2jew2-build / app-root/data/jenkins/workspace/q2jew2-build - hudson.remoting.Channel@197ee95:q2jew2bldr
Using strategy: Default
Checkout:q2jew2-build / app-root/data/jenkins/workspace/q2jew2-build - hudson.remoting.LocalChannel@aae030
Cloning the remote Git repository
Cloning repository origin
Fetching upstream changes from ssh://5195961b548dd90a47000005@q2jew2-qgong2.dev.rhcloud.com/~/git/q2jew2.git/
Seen branch in repository origin/HEAD
Seen branch in repository origin/master
Commencing build of Revision 0b40af17056c3928e14fffff8afd8199bb698c43 (origin/HEAD, origin/master)
Checking out Revision 0b40af17056c3928e14fffff8afd8199bb698c43 (origin/HEAD, origin/master)
No change to record in branch origin/HEAD
No change to record in branch origin/master
[q2jew2-build] $ /bin/sh -xe /tmp/hudson648486737048466732.sh
+ source /usr/libexec/openshift/cartridges/abstract/info/lib/jenkins_util
+ jenkins_rsync '5195961b548dd90a47000005@q2jew2-qgong2.dev.rhcloud.com:~/.m2/' /var/lib/openshift/968688419180368583720960//.m2/
+ rsync --delete-after -az -e /usr/libexec/openshift/cartridges/v2/jenkins/bin/git_ssh_wrapper.sh '5195961b548dd90a47000005@q2jew2-qgong2.dev.rhcloud.com:~/.m2/' /var/lib/openshift/968688419180368583720960//.m2/
+ . ci_build.sh
/tmp/hudson648486737048466732.sh: line 7: .: ci_build.sh: file not found
Build step 'Execute shell' marked build as failure
Archiving artifacts
Finished: FAILURE
Comment 9 Rony Gong 2013-05-17 04:55:27 EDT
comment 8's additional step:
6. Rename existing Jenkins jobs
access <jenkins-server-url>/job/$appname-build/configure, rename the value of "Project name"
7, then build, still show that error.
Comment 10 Dan Mace 2013-05-17 11:05:35 EDT
The jenkins-client cartridges were not migrating successfully and would cause the entire gear to be incompletely migrated.

Resolved by https://github.com/openshift/li/pull/1409 and https://github.com/openshift/origin-server/pull/2532/files.

Also, please ensure the node is in v2 mode and the old job has been renamed and the v2 jenkins-client re-added to the app prior to re-testing the job.
Comment 11 openshift-github-bot 2013-05-17 13:02:41 EDT
Commit pushed to master at https://github.com/openshift/li

https://github.com/openshift/li/commit/7fe0643382e58efb54c139909cb49258a4e849d2
Bug 962324: Fix jenkins-client migration
Comment 12 openshift-github-bot 2013-05-17 13:02:45 EDT
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/90da565fdb727b2bfc9f73fda8e852d3bef0dc37
Bug 962324: Add status output to jenkins-client
Comment 13 Wei Sun 2013-05-20 03:50:46 EDT
Veirified.
Upgrading devenv-stage_353 to devenv_3247
Step:
1. Launch devenv-stage_353
2. Create an application with jenkins-client-1.4
3. Add V1 marker to node
#touch /var/lib/openshift/.settings/v1_cartridge_format
4. scp -r ~/devenv-local and /etc/yum.repos.d/* from devenv_3247 instance to same location on devenv-stage_353 instance
5. yum update -y --enablerepo devenv-local
6. Run oo-admin-clear-pending-ops
7. Remove the v1 marker from the node
#rm -f /var/lib/openshift/.settings/v1_cartridge_format
8. Restart mcollective
#service mcollective restart
9. Clear the broker cache
#oo-admin-broker-cache --clear --console
10. Restart the rhc-broker
#server rhc-broker restart
12. Run migrate-mongo-2.0.28 
13. Run rhc-admin-migrate --version 2.0.28
14.Rename existing Jenkins jobs
15.Re-add jenkins-client-1.4 to the app
Result:
[sunwei@dhcp-8-229 r9]$ gitpush 
[master d666b55] test
 1 file changed, 1 insertion(+), 1 deletion(-)
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 276 bytes, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Executing Jenkins build.
remote: 
remote: You can track your build at https://jks-wsun3246mig.dev.rhcloud.com/job/r9-build
remote: 
remote: Waiting for build to schedule............................................Done
remote: Waiting for job to complete.................................................................................Done
remote: SUCCESS
remote: New build has been deployed.
To ssh://51998e6bdb86250a92000001@r9-wsun3246mig.dev.rhcloud.com/~/git/r9.git/
   c6687a2..d666b55  master -> master
Comment 14 Meng Bo 2013-05-21 04:59:43 EDT
@Jhon, reply to comment#1,

How will users get such information after migration?

We cannot see these introduction after finished migration and trying to build with jenkins from Client side.
Comment 15 Jhon Honce 2013-05-21 11:09:57 EDT
The information is being shared with users via blogs, news letters and emails.

Note You need to log in before you can comment on or make changes to this bug.