Bug 1005123

Summary:	Jenkins build fail to detect force_clean_build for ruby apps
Product:	OpenShift Online	Reporter:	Zhe Wang <zhewang>
Component:	Containers	Assignee:	Andy Goldstein <agoldste>
Status:	CLOSED CURRENTRELEASE	QA Contact:	libra bugs <libra-bugs>
Severity:	medium	Docs Contact:
Priority:	medium
Version:	2.x	CC:	agoldste, dmcphers, lzhang, xtian
Target Milestone:	---	Keywords:	Reopened
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2014-02-26 19:07:56 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Bug Depends On:	1020188
Bug Blocks:

Description Zhe Wang 2013-09-06 09:14:45 UTC

Description of problem:
Given a ruby app (either 1.8 or 1.9), when adding a Jenkins client to this app and create a force_clean_build marker, the Jenkins builds of this app always fail to detect the presence of the force_clean_build marker. On the other hand, the force_clean_build works after removing the Jenkins client for the ruby app.

Version-Release number of selected component (if applicable):
INT(devenv_3479)

How reproducible:
always

Steps to Reproduce:
1. create a ruby app (either 1.8 or 1.9) with a Jenkins client cartridge added
rhc app create rb9 ruby-1.9 --enable-jenkins

2. install a ref package to this app with the following Gemfile and Gemfile.lock

Gemfile
<----->
source 'http://rubygems.org'

gem 'ref'
gem 'rack'
<----->

Gemfile.lock
<----->
GEM
  remote: http://rubygems.org/
    specs:
      rack (1.5.2)
      ref (1.0.5)
 
PLATFORMS
  ruby

DEPENDENCIES
  rack
  ref
<----->

3. push the changes and trigger a Jenkins build

4. check the jenkins build log to see if the ref package is installed

5. touch a force_clean_build marker

6. push the change and trigger another Jenkins build

7. check the Jenkins log of the build in Step 6, to see if the ref package is reinstalled

8. remove the Jenkins-client from the ruby app

9. make another change and push again

Actual results:
In Step 4, the ref package was installed successfully, with the following string in the jenkins log

Installing ref (1.0.5)

However, the force_clean_build does not work in Step 6, and when checking the Jenkins log in Step 7, the ref package was not reinstalled, instead, we can see 

Using ref (1.0.5)

in the Jenkins log.

Moreover, after removing the Jenkins-client in Step 8, we can see the ref pkf was reinstalled in Step 9:
remote: Installing rack (1.5.2) 
remote: Installing ref (1.0.5) 
remote: Using bundler (1.1.4) 

Expected results:
Jenkins build with force_clean_build for ruby apps should be successful.

Additional info:

Comment 1 Hiro Asari 2013-09-10 17:15:22 UTC

The fix should look something like this: https://github.com/openshift/origin-server/pull/3602

This will not affect the existing jobs on the end users' Jenkins gear.

I am assigning this to Paul, who is reworking this logic.

Comment 2 Paul Morie 2013-10-17 21:37:52 UTC


*** This bug has been marked as a duplicate of bug 1020188 ***

Comment 3 Zhe Wang 2013-10-21 08:48:06 UTC

The problem that force_clean_build does not work with Jenkins builds for ALL carts have been fixed in Bug 1020188 (Sprint 35). But this one, Bug 1005123, for ruby apps only still exits in STG(devenv-stage_521) (Sprint 34).

Reopen this bug to track the fix in STG of Sprint 34.

Comment 4 Paul Morie 2013-10-21 14:48:09 UTC

This is fixed in sprint 35.  We will not fix this in .34.

Comment 5 Zhe Wang 2013-10-22 06:48:19 UTC

Move this bug to VERIFIED referring to Comment#3 & #4.

Comment 6 Lei Zhang 2014-01-23 11:48:41 UTC

This issue can be reproduced on devenv_4261, so reopen this bug

1. git push with jenkins-client, force_clean_build doesn't work

Force clean build enabled - cleaning dependencies
Building git ref 'origin/HEAD', commit 9b0be3b
ls -la /var/lib/openshift/52e0fbb063b19a4e880005f1/app-root/runtime/repo/vendor/bundle/ruby/1.9*/gems/ref*
ls: cannot access /var/lib/openshift/52e0fbb063b19a4e880005f1/app-root/runtime/repo/vendor/bundle/ruby/1.9*/gems/ref*: No such file or directory
RESULT=1
Building Ruby cartridge
NOTE: Skipping 'bundle install' because Gemfile is not modified.
+ /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh 52e0fac063b19a4e880005ce.rhcloud.com 'gear stop --conditional --exclude-web-proxy --git-ref 9b0be3bf6fd3a50b399d4841d88d1bf0fc5ee165'
Stopping gear...
Stopping Ruby cartridge
Waiting for stop to finish
++ /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh 52e0fac063b19a4e880005ce.rhcloud.com 'gear create-deployment-dir'
+ deployment_dir=2014-01-23_06-31-24.440
+ rsync --delete-after -az -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh /var/lib/openshift/52e0fbb063b19a4e880005f1//app-deployments/current/metadata.json 52e0fac063b19a4e880005ce.rhcloud.com:app-deployments/2014-01-23_06-31-24.440/metadata.json
+ rsync --delete-after -az -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh --exclude .git /var/lib/openshift/52e0fbb063b19a4e880005f1/app-root/runtime/repo/ 52e0fac063b19a4e880005ce.rhcloud.com:app-root/runtime/repo/
+ rsync --delete-after -az -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh /var/lib/openshift/52e0fbb063b19a4e880005f1/app-root/runtime/build-dependencies/ 52e0fac063b19a4e880005ce.rhcloud.com:app-root/runtime/build-dependencies/
+ rsync --delete-after -az -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh /var/lib/openshift/52e0fbb063b19a4e880005f1/app-root/runtime/dependencies/ 52e0fac063b19a4e880005ce.rhcloud.com:app-root/runtime/dependencies/
+ /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh 52e0fac063b19a4e880005ce.rhcloud.com 'gear remotedeploy --deployment-datetime 2014-01-23_06-31-24.440'
Preparing build for deployment
Deployment id is 93f81a37
Activating deployment
ls -la /var/lib/openshift/52e0fac063b19a4e880005ce/app-root/runtime/repo/vendor/bundle/ruby/1.9*/gems/ref*
ls: cannot access /var/lib/openshift/52e0fac063b19a4e880005ce/app-root/runtime/repo/vendor/bundle/ruby/1.9*/gems/ref*: No such file or directory
RESULT=1
Starting Ruby cartridge
Archiving artifacts
Finished: SUCCESS

2. git push after remove jenkins-client, force_clean_build works well

remote: Stopping Ruby cartridge
remote: [Thu Jan 23 06:29:43 2014] [warn] PassEnv variable OPENSHIFT_JENKINS_CLIENT_IDENT was undefined
remote: [Thu Jan 23 06:29:43 2014] [warn] PassEnv variable OPENSHIFT_JENKINS_CLIENT_DIR was undefined
remote: Waiting for stop to finish
remote: Force clean build enabled - cleaning dependencies
remote: Building git ref 'master', commit e120b2d
remote: ls -la /var/lib/openshift/52e0fac063b19a4e880005ce/app-root/runtime/repo/vendor/bundle/ruby/1.9*/gems/ref*
remote: ls: cannot access /var/lib/openshift/52e0fac063b19a4e880005ce/app-root/runtime/repo/vendor/bundle/ruby/1.9*/gems/ref*: No such file or directory
remote: RESULT=1
remote: Building Ruby cartridge
remote: bundle install --deployment --path ./app-root/repo/vendor/bundle
remote: Fetching gem metadata from http://rubygems.org/..........
remote: Installing rack (1.5.2) 
remote: Installing ref (1.0.5) 
remote: Using bundler (1.1.4) 
remote: Your bundle is complete! It was installed into ./vendor/bundle
remote: Preparing build for deployment
remote: Deployment id is 8d9b7b98
remote: Activating deployment
remote: ls -la /var/lib/openshift/52e0fac063b19a4e880005ce/app-root/runtime/repo/vendor/bundle/ruby/1.9*/gems/ref*
remote: total 32
remote: drwx------. 5 52e0fac063b19a4e880005ce 52e0fac063b19a4e880005ce 4096 Jan 23 06:29 .
remote: drwx------. 4 52e0fac063b19a4e880005ce 52e0fac063b19a4e880005ce 4096 Jan 23 06:29 ..
remote: -rw-r--r--. 1 52e0fac063b19a4e880005ce 52e0fac063b19a4e880005ce 1056 Jan 23 06:29 MIT_LICENSE
remote: -rw-r--r--. 1 52e0fac063b19a4e880005ce 52e0fac063b19a4e880005ce 3426 Jan 23 06:29 README.rdoc
remote: -rw-r--r--. 1 52e0fac063b19a4e880005ce 52e0fac063b19a4e880005ce    5 Jan 23 06:29 VERSION
remote: drwx------. 3 52e0fac063b19a4e880005ce 52e0fac063b19a4e880005ce 4096 Jan 23 06:29 ext
remote: drwx------. 4 52e0fac063b19a4e880005ce 52e0fac063b19a4e880005ce 4096 Jan 23 06:29 lib
remote: drwx------. 2 52e0fac063b19a4e880005ce 52e0fac063b19a4e880005ce 4096 Jan 23 06:29 test
remote: RESULT=0
remote: Starting Ruby cartridge
remote: -------------------------
remote: Git Post-Receive Result: success
remote: Activation status: success
remote: Deployment completed with status: success
To ssh://52e0fac063b19a4e880005ce.rhcloud.com/~/git/myruby19.git/
   deb46f9..e120b2d  master -> master

Comment 7 Andy Goldstein 2014-01-23 16:36:14 UTC

https://github.com/openshift/origin-server/pull/4572

Comment 8 openshift-github-bot 2014-01-23 19:30:15 UTC

Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/fa62069ff0f322d16e15650d73bb4c7166f67e3c
Bug 1005123

Move deletion of checksum file for Gemfile.lock from pre-repo-archive to
build because pre-repo-archive isn't invoked when using a builder
cartridge, meaning that force_clean_build wouldn't always be respected.

Update the checksum file if it changes, so future checksum comparisons
will evaluate correctly.

Comment 9 openshift-github-bot 2014-01-23 19:37:36 UTC

Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/c0fec2ea60bcbe1721c51acbebd8916dc2bf075d
Merge branch 'master' of github.com:openshift/origin-server

* 'master' of github.com:openshift/origin-server:
  Bug 1005123

Comment 10 Lei Zhang 2014-01-24 06:22:13 UTC

Verified on INT(devenv_4267)

Building remotely on app1bldr in workspace /var/lib/openshift/52e20526ba29cbb5570000b4/app-root/runtime/repo Checkout:repo / /var/lib/openshift/52e20526ba29cbb5570000b4/app-root/runtime/repo - hudson.remoting.Channel@1a42cfd:app1bldr Using strategy: Default Last Built Revision: Revision 9e30fb42e29ce57007fc6fc373296b39464cf3a5 (origin/HEAD, origin/master) Checkout:repo / /var/lib/openshift/52e20526ba29cbb5570000b4/app-root/runtime/repo - hudson.remoting.LocalChannel@8b1b21 Wiping out workspace first. Cloning the remote Git repository Cloning repository origin Fetching upstream changes from ssh://52e204dcba29cbb557000097.rhcloud.com/~/git/app1.git Seen branch in repository origin/HEAD Seen branch in repository origin/master Commencing build of Revision 292fd8e1ea6e79f7629e206e7d0b5c61eb979689 (origin/HEAD, origin/master) Checking out Revision 292fd8e1ea6e79f7629e206e7d0b5c61eb979689 (origin/HEAD, origin/master) Warning : There are multiple branch changesets here [repo] $ /bin/sh -xe /tmp/hudson5165688112189249642.sh + source /usr/lib/openshift/cartridge_sdk/bash/sdk ++ '[' false == true ']' ++ OO_BASH_SDK=true + alias 'rsync=rsync --delete-after -az -e '\''/usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh'\''' + upstream_ssh=52e204dcba29cbb557000097.rhcloud.com + rm -f /var/lib/openshift/52e20526ba29cbb5570000b4//app-deployments/current/metadata.json + marker_present force_clean_build + '[' -f /var/lib/openshift/52e20526ba29cbb5570000b4/app-root/runtime/repo//.openshift/markers/force_clean_build ']' + gear build Force clean build enabled - cleaning dependencies Building git ref 'origin/HEAD', commit 292fd8e ls -la /var/lib/openshift/52e20526ba29cbb5570000b4/app-root/runtime/repo/vendor/bundle/ruby/1.9*/gems/ref* ls: cannot access /var/lib/openshift/52e20526ba29cbb5570000b4/app-root/runtime/repo/vendor/bundle/ruby/1.9*/gems/ref*: No such file or directory RESULT=1 Building Ruby cartridge bundle install --deployment --path ./app-root/repo/vendor/bundle
Fetching gem metadata from http://rubygems.org/.......... Installing rack (1.5.2) Installing ref (1.0.5) Using bundler (1.1.4) Your bundle is complete! It was installed into ./vendor/bundle + /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh 52e204dcba29cbb557000097.rhcloud.com 'gear stop --conditional --exclude-web-proxy --git-ref 292fd8e1ea6e79f7629e206e7d0b5c61eb979689'
Stopping gear... Stopping Ruby cartridge Waiting for stop to finish ++ /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh 52e204dcba29cbb557000097.rhcloud.com 'gear create-deployment-dir'
+ deployment_dir=2014-01-24_01-18-19.133 + rsync --delete-after -az -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh /var/lib/openshift/52e20526ba29cbb5570000b4//app-deployments/current/metadata.json 52e204dcba29cbb557000097.rhcloud.com:app-deployments/2014-01-24_01-18-19.133/metadata.json + rsync --delete-after -az -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh --exclude .git /var/lib/openshift/52e20526ba29cbb5570000b4/app-root/runtime/repo/ 52e204dcba29cbb557000097.rhcloud.com:app-root/runtime/repo/
+ rsync --delete-after -az -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh /var/lib/openshift/52e20526ba29cbb5570000b4/app-root/runtime/build-dependencies/ 52e204dcba29cbb557000097.rhcloud.com:app-root/runtime/build-dependencies/ + rsync --delete-after -az -e /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh /var/lib/openshift/52e20526ba29cbb5570000b4/app-root/runtime/dependencies/ 52e204dcba29cbb557000097.rhcloud.com:app-root/runtime/dependencies/
+ /usr/libexec/openshift/cartridges/jenkins/bin/git_ssh_wrapper.sh 52e204dcba29cbb557000097.rhcloud.com 'gear remotedeploy --deployment-datetime 2014-01-24_01-18-19.133'
Preparing build for deployment Deployment id is 2fb9274b
Activating deployment ls -la /var/lib/openshift/52e204dcba29cbb557000097/app-root/runtime/repo/vendor/bundle/ruby/1.9*/gems/ref* total 32 drwx------. 5 52e204dcba29cbb557000097 52e204dcba29cbb557000097 4096 Jan 24 01:18 . drwx------. 4 52e204dcba29cbb557000097 52e204dcba29cbb557000097 4096 Jan 24 01:18 .. -rw-r--r--. 1 52e204dcba29cbb557000097 52e204dcba29cbb557000097 1056 Jan 24 01:18 MIT_LICENSE -rw-r--r--. 1 52e204dcba29cbb557000097 52e204dcba29cbb557000097 3426 Jan 24 01:18 README.rdoc -rw-r--r--. 1 52e204dcba29cbb557000097 52e204dcba29cbb557000097 5 Jan 24 01:18 VERSION drwx------. 3 52e204dcba29cbb557000097 52e204dcba29cbb557000097 4096 Jan 24 01:18 ext drwx------. 4 52e204dcba29cbb557000097 52e204dcba29cbb557000097 4096 Jan 24 01:18 lib drwx------. 2 52e204dcba29cbb557000097 52e204dcba29cbb557000097 4096 Jan 24 01:18 test RESULT=0 Starting Ruby cartridge
Archiving artifacts Finished: SUCCESS