Bug 1035200

Summary: Fail to git push for app after upgrade ose from 1.2 to 2.0
Product: OpenShift Container Platform Reporter: Johnny Liu <jialiu>
Component: Cluster Version OperatorAssignee: Jason DeTiberus <jdetiber>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 2.0.0CC: bleanhar, gpei, jdetiber, libra-onpremise-devel
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-02-04 14:43:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Johnny Liu 2013-11-27 09:38:57 UTC
Description of problem:
after upgrade to ose-2.0, app fail to git push.

Version-Release number of selected component (if applicable):
openshift-enterprise-upgrade-node-2.0.0b-1.git.16.a0bff65.el6op.noarch
openshift-enterprise-release-2.0.0b-1.git.16.a0bff65.el6op.noarch

How reproducible:
Always

Steps to Reproduce:
1.Setup ose-1.2 env
2.Create an app.
3.Then upgrade env to 2.0.
4.Do some change to this app, then git push

Actual results:
[jialiu@jialiu-pc1 jbeapscal]$ git commit -a -mx; git push
[master 6fd9139] x
 1 file changed, 1 insertion(+), 1 deletion(-)
<--snip-->
Counting objects: 5, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 279 bytes | 0 bytes/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote: + exec
remote: Permission denied - /var/lib/openshift/52958455aeb9059466000020/app-deployments
To ssh://52958455aeb9059466000020.com.cn/~/git/jbeapscal.git/
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://52958455aeb9059466000020.com.cn/~/git/jbeapscal.git/'


Expected results:
git push succeed.

Additional info:

Comment 2 Jason DeTiberus 2013-11-28 05:39:00 UTC
This is fixed in the latest pad: http://etherpad.corp.redhat.com/ose-2-0-upgrade-2013-11-27

Comment 3 Gaoyun Pei 2013-11-28 09:22:00 UTC
This bug still could be reproduced with 
openshift-enterprise-release-2.0.0c-1.git.2.acc7a39.el6op.noarch
openshift-enterprise-upgrade-node-2.0.0c-1.git.2.acc7a39.el6op.noarch

1. Setup ose-1.2 env
2. Create a scalable python-2.6 app on the node
3. Upgrade env from 1.2 to 2.0
4.Do some change to this app, then git push
[root@broker app]# git push
Counting objects: 4, done.
Delta compression using up to 2 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 282 bytes, done.
Total 3 (delta 1), reused 0 (delta 0)
remote: undefined method `values' for nil:NilClass
To ssh://529701466892df65da000005.com/~/git/app.git/
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://529701466892df65da000005.com/~/git/app.git/'

Comment 4 Johnny Liu 2013-12-02 06:27:51 UTC
For PHP app, it throw the following error:
$ git commit -a -mx; git push
[master 82f73ee] x
 1 file changed, 1 insertion(+), 1 deletion(-)

Warning: Permanently added 'phpapp-jialiu.my-ose.com.cn,10.4.59.174' (RSA) to the list of known hosts.
Counting objects: 7, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 364 bytes | 0 bytes/s, done.
Total 4 (delta 2), reused 0 (delta 0)
remote: Stopping Cron cartridge
remote: Stopping MySQL cartridge
remote: Stopping PHP cartridge
remote: [Mon Dec 02 01:24:54 2013] [warn] PassEnv variable SHELL was undefined
remote: [Mon Dec 02 01:24:54 2013] [warn] PassEnv variable USER was undefined
remote: [Mon Dec 02 01:24:54 2013] [warn] PassEnv variable LOGNAME was undefined
remote: WARNING: Require ServerLimit > 0, setting to 1
remote: WARNING: Require MaxClients > 0, setting to 1
remote: Waiting for stop to finish
remote: Permission denied - /var/lib/openshift/5295d4a8aeb90571cc00008f/app-deployments
To ssh://5295d4a8aeb90571cc00008f.com.cn/~/git/phpapp.git/
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://5295d4a8aeb90571cc00008f.com.cn/~/git/phpapp.git/'

Comment 5 Jason DeTiberus 2013-12-03 00:14:54 UTC
This is fixed in the latest upgrade puddle: http://etherpad.corp.redhat.com/ose-2-0-upgrade-2013-12-02

Please note that this pad has changed the upgrade steps to use a puddle, so there are a few workarounds that need to be done to make sure the right bits are installed during the upgrade.

I tested with python-2.6, and php both scalable and non-scalable.

Comment 6 Johnny Liu 2013-12-03 09:38:34 UTC
Rest this bug following http://etherpad.corp.redhat.com/ose-2-0-upgrade-2013-12-02, still failed. The same error:
remote: Permission denied - /var/lib/openshift/529d87ccf16d95354f000010/app-deployments

Now I already set up an ose-1.2 test bed for you, I also created all kinds of cartridge app, you could use it for upgrade testing to reproduce this bug.

Broker: 10.4.59.154 root/redhat
Node: 10.4.59.132 root/redhat
Account/password: jialiu/redhat

Comment 7 Jason DeTiberus 2013-12-03 21:40:27 UTC
I updated yesterday's pad later in the evening(EST) to fix for a broken puddle that was built earlier, I think you may have tested against that older puddle.

The new pad/puddle has all the proper packages and fixes a mysql cartridge upgrade bug.
http://etherpad.corp.redhat.com/ose-2-0-upgrade-2013-12-03

Comment 8 Johnny Liu 2013-12-05 09:30:13 UTC
Verified this bug following http://etherpad.corp.redhat.com/ose-2-0-upgrade-2013-12-04, and PASS.