Bug 964180

Summary: Failed to git push a scalable php app with multiple gears after migration
Product: OpenShift Online Reporter: Meng Bo <bmeng>
Component: ContainersAssignee: Mrunal Patel <mpatel>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.xCC: dmace, mpatel
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-06-11 04:05:51 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Meng Bo 2013-05-17 12:35:06 UTC
Description of problem:
Migration scalable app from v1 to v2, try to migration the app after migration. Git push failed.

Version-Release number of selected component (if applicable):
devenv-stage_353 to devenv_3235

How reproducible:
always

Steps to Reproduce:
1. Launch devenv-stage_353
2. Create scalable php app with multiple gears on it
3. Add V1 marker to node
#touch /var/lib/openshift/.settings/v1_cartridge_format
4. scp -r ~/devenv-local and /etc/yum.repos.d/* from devenv_3235 instance to same location on devenv-stage_353 instance
5. yum update -y --enablerepo devenv-local
6. Run oo-admin-clear-pending-ops
7. Remove the v1 marker from the node
#rm -f /var/lib/openshift/.settings/v1_cartridge_format
8. Clear the broker cache
#rake tmp:clear
9. Restart the rhc-broker and mcollective
10. Run migrate-mongo-2.0.28 
11. Run rhc-admin-migrate --version 2.0.28
12. After migration finished, modify the app git repo and git push  

Actual results:
Counting objects: 3, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 239 bytes, done.
Total 2 (delta 1), reused 0 (delta 0)
remote: CLIENT_RESULT: HAProxy instance is stopped
remote: Stopping PHP cart
remote: [Fri May 17 08:03:59 2013] [warn] module php5_module is already loaded, skipping
remote: Stopping PHP cart
remote: [Fri May 17 08:04:00 2013] [warn] module php5_module is already loaded, skipping
remote: httpd (no pid file) not running
remote: HAProxy already stopped
remote: CLIENT_RESULT: HAProxy instance is stopped
remote: Starting PHP cart
remote: [Fri May 17 08:04:25 2013] [warn] module php5_module is already loaded, skipping
remote: Starting PHP cart
remote: [Fri May 17 08:04:26 2013] [warn] module php5_module is already loaded, skipping
remote: httpd (pid 7328) already running
remote: SSH_CMD: ssh 354811445906812415508480.5.196
remote: ++ date
remote: + echo 'Syncing to gear: 354811445906812415508480.5.196:php @ ' Fri May 17 08:04:27 EDT 2013
remote: Syncing to gear: 354811445906812415508480.5.196:php @  Fri May 17 08:04:27 EDT 2013
remote: + for rpccall in '"${OPENSHIFT_SYNC_GEARS_PRE[@]}"'
remote: + ssh 354811445906812415508480.5.196 'gear stop'
remote: Warning: Identity file 354811445906812415508480.5.196 not accessible: No such file or directory.
remote: Exit code: 
remote: An error occured executing the 'postreceive' command. For more details about the problem, try running the command again with the '--trace' option.
To ssh://3b7d8a18be9d11e2b9f122000aa505c4.rhcloud.com/~/git/php1s.git/
   c4964a1..d782ad9  master -> master


Expected results:
Should not have such errors and git push successfully.

Additional info:

Comment 1 Vojtech Vitek 2013-05-17 14:59:16 UTC
The only place where we load php5_module is at:

> openshift-origin-cartridge-php/configuration/etc/conf.d/php.conf
> LoadModule php5_module modules/libphp5.so

I can't quite understand how this module would be "already" loaded into the Apache, which is started using:
> /usr/sbin/httpd -C "Include $HTTPD_CFG_DIR/*.conf" -f $HTTPD_CFG_FILE -k stop

Comment 2 Vojtech Vitek 2013-05-17 15:02:38 UTC
Need to dig into:

remote: An error occured executing the 'postreceive' command. For more details about the problem, try running the command again with the '--trace' option.

Comment 3 Mrunal Patel 2013-05-17 19:56:29 UTC
Fixed in devenv_3236+
by https://github.com/openshift/li/pull/1401

Comment 4 Meng Bo 2013-05-20 07:47:17 UTC
Issue fixed by upgrade to devenv_3247,

Counting objects: 3, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (2/2), 239 bytes, done.
Total 2 (delta 1), reused 0 (delta 0)
remote: Stopping PHP cart
remote: [Mon May 20 03:37:55 2013] [warn] module php5_module is already loaded, skipping
remote: CLIENT_RESULT: HAProxy instance is stopped
remote: CLIENT_RESULT: HAProxy instance is started
remote: SSH_CMD: ssh 996575092668146865143808.15.32
remote: ++ date
remote: + echo 'Syncing to gear: 996575092668146865143808.15.32:php-5.3 @ ' Mon May 20 03:38:18 EDT 2013
remote: Syncing to gear: 996575092668146865143808.15.32:php-5.3 @  Mon May 20 03:38:18 EDT 2013
remote: + for rpccall in '"${OPENSHIFT_SYNC_GEARS_PRE[@]}"'
remote: + ssh 996575092668146865143808.15.32 'gear stop'
remote: Stopping gear...
remote: Stopping PHP cart
remote: [Mon May 20 03:38:31 2013] [warn] module php5_module is already loaded, skipping
remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"'
remote: + '[' -d /var/lib/openshift/51998f7ddb86250715000001/php//../app-root/repo ']'
remote: + rsync -v --delete-after -az /var/lib/openshift/51998f7ddb86250715000001/php//../app-root/repo/ 996575092668146865143808.15.32:php-5.3/../app-root/repo/
remote: building file list ... done
remote: ./
remote: 123
remote: README
remote: deplist.txt
remote: .openshift/
remote: .openshift/action_hooks/
remote: .openshift/action_hooks/build
remote: .openshift/action_hooks/deploy
remote: .openshift/action_hooks/post_deploy
remote: .openshift/action_hooks/post_start_php-5.3
remote: .openshift/action_hooks/post_stop_php-5.3
remote: .openshift/action_hooks/pre_build
remote: .openshift/action_hooks/pre_start_php-5.3
remote: .openshift/action_hooks/pre_stop_php-5.3
remote: .openshift/cron/
remote: .openshift/cron/README.cron
remote: .openshift/cron/daily/
remote: .openshift/cron/daily/.gitignore
remote: .openshift/cron/hourly/
remote: .openshift/cron/hourly/.gitignore
remote: .openshift/cron/minutely/
remote: .openshift/cron/minutely/.gitignore
remote: .openshift/cron/monthly/
remote: .openshift/cron/monthly/.gitignore
remote: .openshift/cron/weekly/
remote: .openshift/cron/weekly/README
remote: .openshift/cron/weekly/chrono.dat
remote: .openshift/cron/weekly/chronograph
remote: .openshift/cron/weekly/jobs.allow
remote: .openshift/cron/weekly/jobs.deny
remote: .openshift/markers/
remote: .openshift/markers/README
remote: libs/
remote: libs/.gitkeep
remote: misc/
remote: misc/.gitkeep
remote: php/
remote: php/health_check.php
remote: php/index.php
remote: 
remote: sent 1771 bytes  received 731 bytes  1668.00 bytes/sec
remote: total size is 14566  speedup is 5.82
remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"'
remote: + '[' -d /var/lib/openshift/51998f7ddb86250715000001/php//node_modules ']'
remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"'
remote: + '[' -d /var/lib/openshift/51998f7ddb86250715000001/php//virtenv ']'
remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"'
remote: + '[' -d /var/lib/openshift/51998f7ddb86250715000001/php//../.m2 ']'
remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"'
remote: + '[' -d /var/lib/openshift/51998f7ddb86250715000001/php//.openshift ']'
remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"'
remote: + '[' -d /var/lib/openshift/51998f7ddb86250715000001/php//deployments ']'
remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"'
remote: + '[' -d /var/lib/openshift/51998f7ddb86250715000001/php//perl5lib ']'
remote: + for subd in '"${OPENSHIFT_SYNC_GEARS_DIRS[@]}"'
remote: + '[' -d /var/lib/openshift/51998f7ddb86250715000001/php//phplib ']'
remote: + rsync -v --delete-after -az /var/lib/openshift/51998f7ddb86250715000001/php//phplib/ 996575092668146865143808.15.32:php-5.3/phplib/
remote: building file list ... done
remote: ./
remote: pear/
remote: pear/cache/
remote: pear/cfg/
remote: pear/data/
remote: pear/docs/
remote: pear/download/
remote: pear/ext/
remote: pear/pear/
remote: pear/pear/php/
remote: pear/pear/php/.depdb
remote: pear/pear/php/.depdblock
remote: pear/pear/php/.filemap
remote: pear/pear/php/.lock
remote: pear/pear/php/.channels/
remote: pear/pear/php/.channels/__uri.reg
remote: pear/pear/php/.channels/doc.php.net.reg
remote: pear/pear/php/.channels/pear.php.net.reg
remote: pear/pear/php/.channels/pecl.php.net.reg
remote: pear/pear/php/.channels/.alias/
remote: pear/pear/php/.channels/.alias/pear.txt
remote: pear/pear/php/.channels/.alias/pecl.txt
remote: pear/pear/php/.channels/.alias/phpdocs.txt
remote: pear/pear/php/.registry/
remote: pear/pear/php/.registry/.channel.__uri/
remote: pear/pear/php/.registry/.channel.doc.php.net/
remote: pear/pear/php/.registry/.channel.pecl.php.net/
remote: pear/php/
remote: pear/temp/
remote: pear/tests/
remote: pear/www/
remote: 
remote: sent 2009 bytes  received 335 bytes  937.60 bytes/sec
remote: total size is 2083  speedup is 0.89
remote: + for rpccall in '"${OPENSHIFT_SYNC_GEARS_POST[@]}"'
remote: + ssh 996575092668146865143808.15.32 'gear remotedeploy'
remote: TODO
remote: Starting PHP cart
remote: [Mon May 20 03:38:47 2013] [warn] module php5_module is already loaded, skipping
remote: Exit code: 0
remote: TODO
remote: Starting PHP cart
remote: [Mon May 20 03:38:50 2013] [warn] module php5_module is already loaded, skipping
To ssh://51998f7ddb86250715000001.rhcloud.com/~/git/php1s.git/
   c4964a1..9a4430e  master -> master