Bug 962324

Message to users on Jenkins migration:

To support future features, during the new cartridge refactor we took the opportunity to make enhancements to our Jenkins support. Unfortunately, these changes require some work on your part.
 
Steps to update Jenkins jobs to work with new cartridge format:

1. Rename your existing Jenkins jobs (This backs up any changes you have made)

2. Remove your existing Jenkins-client cartridges from your applications

3. Re-add the Jenkins-client to your applications (This will create a new job with all it's new goodness)

Step#4 is only required if you have modified the default Jenkins job.

4. Edit the new Jenkins job and enter any custom logic from your renamed old Jenkins jobs

Sorry for the inconvenience, as we continue to strive to bring you the best PaaS we can.

this issue is reproduced on devenv_3223, please refer to the following results:

Steps to Reproduce:
1. Launch a devenv-stage_351 ami
2. Create an app with jenkins-client-1.4 (v1 format)
3. scp -r ~/devenv-local and /etc/yum.repos.d/local.repo from devenv_3219 instance to same location on devenv-stage_351 instance
4. yum -y update --enablerepo devenv-local (the instance will be default to v2 format after upgrade)
5. rhc-v2-migrate --version 2.0.28
6. Rename existing Jenkins jobs
access <jenkins-server-url>/job/$appname-build/configure, rename the value of "Project name"
7. Remove Jenkins-client cartridge from this app
8. Re-add Jenkins-client cartridge to this app
9. Make some changes and Jenkins build

After Step9,met the same error messages as before.

For the jenkins error logs as below:

<-------------snip-------------->
+ /usr/libexec/openshift/cartridges/v2/jenkins/bin/git_ssh_wrapper.sh a841b0c2bc3e11e29a6b22000a880b50.rhcloud.com 'gear stop --conditional'
Stopping gear...
cat: /var/lib/openshift/a841b0c2bc3e11e29a6b22000a880b50/jbossews//run/jboss.pid: No such file or directory
Failed to execute: 'control stop' for /var/lib/openshift/a841b0c2bc3e11e29a6b22000a880b50/jbossews: rc(1)
stdout: #<IO:0x00000001117398>
stderr#<IO:0x00000001117320>
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/v2_cart_model.rb:950:in `block in do_control_with_directory'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/v2_cart_model.rb:776:in `process_cartridges'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/v2_cart_model.rb:923:in `do_control_with_directory'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/v2_cart_model.rb:799:in `do_control'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/v2_cart_model.rb:1156:in `stop_cartridge'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/v2_cart_model.rb:1023:in `block in stop_gear'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/v2_cart_model.rb:78:in `block in each_cartridge'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/v2_cart_model.rb:783:in `block in process_cartridges'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/v2_cart_model.rb:780:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/v2_cart_model.rb:780:in `process_cartridges'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/v2_cart_model.rb:76:in `each_cartridge'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/v2_cart_model.rb:1022:in `stop_gear'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.1/lib/openshift-origin-node/model/application_container.rb:324:in `stop_gear'
/usr/bin/gear:210:in `block (3 levels) in <main>'
/usr/bin/gear:80:in `do_command'
/usr/bin/gear:202:in `block (2 levels) in <main>'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:180:in `call'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/command.rb:155:in `run'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:385:in `run_active_command'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/runner.rb:62:in `run!'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/delegates.rb:11:in `run!'
/opt/rh/ruby193/root/usr/share/gems/gems/commander-4.0.3/lib/commander/import.rb:10:in `block in <top (required)>'
Build step 'Execute shell' marked build as failure
Archiving artifacts
Finished: FAILURE

Dan,

  This looks like ews doesn't handle the pid file not being in place.

cat: /var/lib/openshift/a841b0c2bc3e11e29a6b22000a880b50/jbossews//run/jboss.pid: No such file or directory
Failed to execute: 'control stop' for /var/lib/openshift/a841b0c2bc3e11e29a6b22000a880b50/jbossews: rc(1)


It may need to be fixed during the migration but should probably also be handled by the control script in general.

Should be resolved by https://github.com/openshift/origin-server/pull/2507 and https://github.com/openshift/li/pull/1390 (although the stack traces are another issue covered by bug 960809).

Commit pushed to master at https://github.com/openshift/li

https://github.com/openshift/li/commit/da7e991e703d24d3a7f676d435665be9dbba9049
Bug 962324: Fix jbossews stop logic to tolerate missing pidfile

Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/26f7bed6eef569302eefd578d301a0c502310df5
Bug 962324: Fix jbossews stop logic to tolerate missing pidfile

Retest on migration to devenv_3235, Failed
Steps:
1. create diy, jbossews-2.0 app and embed jenkins-client
2. do migration
3. do change to this application, then git push, jenkins build trigged, Failed with jenkins log:

Started by user Jenkins System Builder
Building remotely on q2jew2bldr in workspace app-root/data/jenkins/workspace/q2jew2-build
Checkout:q2jew2-build / app-root/data/jenkins/workspace/q2jew2-build - hudson.remoting.Channel@197ee95:q2jew2bldr
Using strategy: Default
Checkout:q2jew2-build / app-root/data/jenkins/workspace/q2jew2-build - hudson.remoting.LocalChannel@aae030
Cloning the remote Git repository
Cloning repository origin
Fetching upstream changes from ssh://5195961b548dd90a47000005.rhcloud.com/~/git/q2jew2.git/
Seen branch in repository origin/HEAD
Seen branch in repository origin/master
Commencing build of Revision 0b40af17056c3928e14fffff8afd8199bb698c43 (origin/HEAD, origin/master)
Checking out Revision 0b40af17056c3928e14fffff8afd8199bb698c43 (origin/HEAD, origin/master)
No change to record in branch origin/HEAD
No change to record in branch origin/master
[q2jew2-build] $ /bin/sh -xe /tmp/hudson648486737048466732.sh
+ source /usr/libexec/openshift/cartridges/abstract/info/lib/jenkins_util
+ jenkins_rsync '5195961b548dd90a47000005.rhcloud.com:~/.m2/' /var/lib/openshift/968688419180368583720960//.m2/
+ rsync --delete-after -az -e /usr/libexec/openshift/cartridges/v2/jenkins/bin/git_ssh_wrapper.sh '5195961b548dd90a47000005.rhcloud.com:~/.m2/' /var/lib/openshift/968688419180368583720960//.m2/
+ . ci_build.sh
/tmp/hudson648486737048466732.sh: line 7: .: ci_build.sh: file not found
Build step 'Execute shell' marked build as failure
Archiving artifacts
Finished: FAILURE

comment 8's additional step:
6. Rename existing Jenkins jobs
access <jenkins-server-url>/job/$appname-build/configure, rename the value of "Project name"
7, then build, still show that error.

The jenkins-client cartridges were not migrating successfully and would cause the entire gear to be incompletely migrated.

Resolved by https://github.com/openshift/li/pull/1409 and https://github.com/openshift/origin-server/pull/2532/files.

Also, please ensure the node is in v2 mode and the old job has been renamed and the v2 jenkins-client re-added to the app prior to re-testing the job.

Commit pushed to master at https://github.com/openshift/li

https://github.com/openshift/li/commit/7fe0643382e58efb54c139909cb49258a4e849d2
Bug 962324: Fix jenkins-client migration

Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/90da565fdb727b2bfc9f73fda8e852d3bef0dc37
Bug 962324: Add status output to jenkins-client

Veirified.
Upgrading devenv-stage_353 to devenv_3247
Step:
1. Launch devenv-stage_353
2. Create an application with jenkins-client-1.4
3. Add V1 marker to node
#touch /var/lib/openshift/.settings/v1_cartridge_format
4. scp -r ~/devenv-local and /etc/yum.repos.d/* from devenv_3247 instance to same location on devenv-stage_353 instance
5. yum update -y --enablerepo devenv-local
6. Run oo-admin-clear-pending-ops
7. Remove the v1 marker from the node
#rm -f /var/lib/openshift/.settings/v1_cartridge_format
8. Restart mcollective
#service mcollective restart
9. Clear the broker cache
#oo-admin-broker-cache --clear --console
10. Restart the rhc-broker
#server rhc-broker restart
12. Run migrate-mongo-2.0.28 
13. Run rhc-admin-migrate --version 2.0.28
14.Rename existing Jenkins jobs
15.Re-add jenkins-client-1.4 to the app
Result:
[sunwei@dhcp-8-229 r9]$ gitpush 
[master d666b55] test
 1 file changed, 1 insertion(+), 1 deletion(-)
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 276 bytes, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: Executing Jenkins build.
remote: 
remote: You can track your build at https://jks-wsun3246mig.dev.rhcloud.com/job/r9-build
remote: 
remote: Waiting for build to schedule............................................Done
remote: Waiting for job to complete.................................................................................Done
remote: SUCCESS
remote: New build has been deployed.
To ssh://51998e6bdb86250a92000001.rhcloud.com/~/git/r9.git/
   c6687a2..d666b55  master -> master

@Jhon, reply to comment#1,

How will users get such information after migration?

We cannot see these introduction after finished migration and trying to build with jenkins from Client side.

The information is being shared with users via blogs, news letters and emails.

https://www.openshift.com/blogs/important-version-two-cartidge-information-for-jenkins-users