Bug 1023345

Summary: Failed to git push existing app after migration
Product: OpenShift Online Reporter: Wei Sun <wsun>
Component: ContainersAssignee: Andy Goldstein <agoldste>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 2.xCC: yadu
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-01-24 03:26:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Wei Sun 2013-10-25 09:00:24 UTC
Description of problem:
Try to git push existing scaling app after migration,but it failed.There are some error messages:
remote: 5260d3fc03ef64dbb500011d (Error activating gear: Shell command '/usr/bin/rsync -av --delete /var/lib/openshift/5260d3fc03ef64dbb500011d/app-deployments/2013-10-25_03-50-29.651/dependencies/ /var/lib/openshift/5260d3fc03ef64dbb500011d/app-root/runtime/dependencies/' returned an error. rc=23)
remote: 5260d36003ef64dbb50000a5 (Error activating gear: Shell command '/usr/bin/rsync -av --delete /var/lib/openshift/5260d36003ef64dbb50000a5/app-deployments/2013-10-25_03-50-29.651/dependencies/ /var/lib/openshift/5260d36003ef64dbb50000a5/app-root/runtime/dependencies/' returned an error. rc=23)


Version-Release number of selected component (if applicable):
INT(devenv_3942)

How reproducible:
Always

Steps to Reproduce:
1.Create a scaling app before server upgrade
2.INT upgrade
3.Change something and git push for existing scaling app

Actual results:
remote: Stopping Ruby cartridge
remote: [Fri Oct 25 03:50:21 2013] [warn] PassEnv variable SHELL was undefined
remote: [Fri Oct 25 03:50:21 2013] [warn] PassEnv variable USER was undefined
remote: [Fri Oct 25 03:50:21 2013] [warn] PassEnv variable LOGNAME was undefined
remote: httpd (no pid file) not running
remote: Syncing git content to other proxy gears
remote: Building git ref 'master', commit fdb894e
remote: Building Ruby cartridge
remote: Preparing build for deployment
remote: Deployment id is 62ac31cd
remote: Distributing deployment to child gears
remote: Activating deployment
remote: Result: failure
remote: Distribution status: success
remote: Activation status: failure
remote: Activation failed for the following gears:
remote: 5260d3fc03ef64dbb500011d (Error activating gear: Shell command '/usr/bin/rsync -av --delete /var/lib/openshift/5260d3fc03ef64dbb500011d/app-deployments/2013-10-25_03-50-29.651/dependencies/ /var/lib/openshift/5260d3fc03ef64dbb500011d/app-root/runtime/dependencies/' returned an error. rc=23)
remote: 5260d36003ef64dbb50000a5 (Error activating gear: Shell command '/usr/bin/rsync -av --delete /var/lib/openshift/5260d36003ef64dbb50000a5/app-deployments/2013-10-25_03-50-29.651/dependencies/ /var/lib/openshift/5260d36003ef64dbb50000a5/app-root/runtime/dependencies/' returned an error. rc=23)
remote: Deployment completed with status: failure
remote: postreceive failed
To ssh://5260d36003ef64dbb50000a5.rhcloud.com/~/git/r8s.git/
   2544867..fdb894e  master -> master


Expected results:
Could git push successfully

Additional info:

Comment 1 Wei Sun 2013-10-29 06:21:23 UTC
Tested this bug in INT(devenv_3957),still could reproduce this bug.

Result:
remote: Preparing build for deployment
remote: Deployment id is 503f8c31
remote: Activating deployment
remote: Result: failure
remote: Activation status: failure
remote: Activation failed for the following gears:
remote: 5264ff9003ef648bd200006a (Error activating gear: Shell command '/usr/bin/rsync -av --delete /var/lib/openshift/5264ff9003ef648bd200006a/app-deployments/2013-10-29_02-05-44.646/dependencies/ /var/lib/openshift/5264ff9003ef648bd200006a/app-root/runtime/dependencies/' returned an error. rc=23)
remote: Deployment completed with status: failure
remote: postreceive failed
To ssh://5264ff9003ef648bd200006a.rhcloud.com/~/git/pythons.git/
   6ac0c37..9f5ce04  master -> master

Comment 2 Yan Du 2013-10-29 11:18:21 UTC
met same error when git push the non-scalable php/ruby app after upgrade devenv_stage_528 to devenv_3959


remote: Stopping PHP cartridge
remote: [Tue Oct 29 07:06:33 2013] [warn] PassEnv variable SHELL was undefined
remote: [Tue Oct 29 07:06:33 2013] [warn] PassEnv variable USER was undefined
remote: [Tue Oct 29 07:06:33 2013] [warn] PassEnv variable LOGNAME was undefined
remote: Waiting for stop to finish
remote: Stopping MySQL cartridge
remote: Building git ref 'master', commit 6aa0dfc
remote: Building PHP cartridge
remote: Preparing build for deployment
remote: Deployment id is 5d8d93d3
remote: Activating deployment
remote: Result: failure
remote: Activation status: failure
remote: Activation failed for the following gears:
remote: 526f72b496300ee01f00001e (Error activating gear: Shell command '/usr/bin/rsync -av --delete /var/lib/openshift/526f72b496300ee01f00001e/app-deployments/2013-10-29_07-06-36.187/build-dependencies/ /var/lib/openshift/526f72b496300ee01f00001e/app-root/runtime/build-dependencies/' returned an error. rc=23)
remote: Deployment completed with status: failure
remote: postreceive failed
To ssh://526f72b496300ee01f00001e.rhcloud.com/~/git/wordpress.git/
   c3ca08e..6aa0dfc  master -> master

Comment 3 Andy Goldstein 2013-10-29 21:02:53 UTC
https://github.com/openshift/li/pull/2059

Comment 4 openshift-github-bot 2013-10-30 00:17:55 UTC
Commit pushed to master at https://github.com/openshift/li

https://github.com/openshift/li/commit/5ff4dc1e2b97ea3fd7bd59a73876fc7278c7970e
Bug 1023345

Make sure app-root/runtime/build-dependencies and dependencies
directories get rw permissions so the gear user can modify the
contents during activation.

Make sure app-root/dependencies and build-depencencies symlinks have
correct permissions as well.

Comment 5 Wei Sun 2013-10-30 05:10:18 UTC
Verified on devenv_3966 which is upgraded from devenv_stage_528.

Result:
remote: Stopping NodeJS cartridge
remote: Syncing git content to other proxy gears
remote: Saving away previously installed Node modules
remote: Building git ref 'master', commit f68a004
remote: Building NodeJS cartridge
remote: npm info it worked if it ends with ok
remote: npm info using npm.37
remote: npm info using node.20
remote: npm info preinstall OpenShift-Sample-App.0
remote: npm info build /var/lib/openshift/52707e81d8e8a717940000bb/app-root/runtime/repo
remote: npm info linkStuff OpenShift-Sample-App.0
remote: npm info install OpenShift-Sample-App.0
remote: npm info postinstall OpenShift-Sample-App.0
remote: npm info ok 
remote: Preparing build for deployment
remote: Deployment id is c5a4c807
remote: Activating deployment
remote: HAProxy already running
remote: HAProxy instance is started
remote: Starting NodeJS cartridge
remote: Result: success
remote: Activation status: success
remote: Deployment completed with status: success
To ssh://52707e81d8e8a717940000bb.rhcloud.com/~/git/n6s.git/
   7f21639..f68a004  master -> master