Bug 1058251

Summary:	deployment roll back does not sync contents to every web gears
Product:	OpenShift Online	Reporter:	Johnny Liu <jialiu>
Component:	oc	Assignee:	Fabiano Franz <ffranz>
Status:	CLOSED CURRENTRELEASE	QA Contact:	libra bugs <libra-bugs>
Severity:	medium	Docs Contact:
Priority:	high
Version:	2.x	CC:	agoldste, bmeng, dmcphers, wsun
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	Bug Fix
Doc Text:		Story Points:	---
Clone Of:
Clones:	1063769 (view as bug list)		Environment:
Last Closed:	2014-02-26 19:10:35 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1063769

Description Johnny Liu 2014-01-27 10:50:14 UTC

Description of problem:
When doing deployment roll back for a scalable app that have multiple web gears, not every web gear is rolled back, only head web gear does.

Version-Release number of selected component (if applicable):
devenv_stage-655

How reproducible:
Always

Steps to Reproduce:
1. Create a scalable app and set min gear to 2.
# rhc app-show phps --gears
ID State Cartridges Size SSH URL
------------------------ ------- ------------------- ----- ----------------------------------------------------------------------------
52e630235babf124c1000003 started php-5.3 haproxy-1.4 small 52e630235babf124c1000003.rhcloud.com
859079266427041736556544 started php-5.3 haproxy-1.4 small 859079266427041736556544.rhcloud.com
2. Configure app to enable keep-deployments
# rhc app-configure phps --no-auto-deploy --keep-deployments 5
3. Did some change to app web page, and git push
# vim php/index.php
# git add .
# git commit -am 'commit1'
[master 22b2fdb] commit1
<--snip-->
4. Manually deploy the above commit.
# rhc deploy 22b2fdb -a phps
Deployment of git ref '22b2fdb' in progress for application phps ...
Stopping PHP 5.3 cartridge (Apache+mod_php)
[Mon Jan 27 05:16:02 2014] [warn] PassEnv variable SHELL was undefined
[Mon Jan 27 05:16:02 2014] [warn] PassEnv variable USER was undefined
[Mon Jan 27 05:16:02 2014] [warn] PassEnv variable LOGNAME was undefined
Waiting for stop to finish
Syncing git content to other proxy gears
Building git ref '22b2fdb', commit 22b2fdb
Checking deplist.txt for PEAR dependency..
Preparing build for deployment
Deployment id is 63da69cc
Distributing deployment to child gears
Activating deployment
HAProxy already running
HAProxy instance is started
Starting PHP 5.3 cartridge (Apache+mod_php)
Success
5. Roll back the change to old one
# rhc deployments phps
5:09 AM, deployment 8a4e7e84
5:16 AM, deployment 63da69cc
# rhc deployment-activate 8a4e7e84 -a phps
Activating deployment '8a4e7e84' on application phps ...
Activating deployment
Stopping PHP 5.3 cartridge (Apache+mod_php)
Waiting for stop to finish
HAProxy already running
HAProxy instance is started
Starting PHP 5.3 cartridge (Apache+mod_php)
Success
6. Check page of every gears for this app.
# curl phps-wsunstg655.dev.rhcloud.com
# curl 859079266427041736556544-wsunstg655.dev.rhcloud.com

Actual results:
Only head web gear (phps-wsunstg655.dev.rhcloud.com) is rolled back, the other one (859079266427041736556544-wsunstg655.dev.rhcloud.com) dose NOT.

Expected results:
When rolling back, every web gear should be rolled back.

Additional info:

Comment 1 Johnny Liu 2014-01-27 11:07:22 UTC

Client Version: rhc-1.19.4

After enable debug options at client side, seem like rhc does not call "gear remotedeploy" in head gear.

# rhc deployment-activate 8a4e7e84 -a phps -d
DEBUG: Using config file /root/.openshift/express.conf
DEBUG: Git config 'git config --get rhc.app-id' returned '52e630235babf124c1000003'
DEBUG: Git config 'git config --get rhc.app-name' returned 'phps'
DEBUG: Git config 'git config --get rhc.domain-name' returned 'wsunstg655'
DEBUG: Authenticating with RHC::Auth::Token
DEBUG: Connecting to https://localhost/broker/rest/api
DEBUG: Client supports API versions 1.1, 1.2, 1.3, 1.4, 1.5, 1.6
DEBUG: Using token authentication
DEBUG: Created new httpclient
DEBUG: Request GET https://localhost/broker/rest/api
DEBUG: SSL Verification failed -- Using self signed cert
DEBUG:    code 200   60 ms
DEBUG: Server supports API versions 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6
DEBUG:    Using API version 1.6
DEBUG: Client API version 1.6 is not current. Refetching API
DEBUG: Using token authentication
DEBUG: Request GET https://localhost/broker/rest/api
DEBUG:    code 200   39 ms
DEBUG: Using token authentication
DEBUG: Request GET https://localhost/broker/rest/domains/wsunstg655/applications/phps
DEBUG:    code 200   79 ms
Activating deployment '8a4e7e84' on application phps ...
DEBUG: Opening Net::SSH connection to phps-wsunstg655.dev.rhcloud.com, 52e630235babf124c1000003, gear activate 8a4e7e84
Activating deployment
Stopping PHP 5.3 cartridge (Apache+mod_php)
Waiting for stop to finish
HAProxy already running
HAProxy instance is started
Starting PHP 5.3 cartridge (Apache+mod_php)
DEBUG: Terminating ...
Success


Then I log into head gear, run "gear remotedeploy" manually.
[phps-wsunstg655.dev.rhcloud.com 52e630235babf124c1000003]\> gear remotedeploy
Preparing build for deployment
Deployment id is cf80cb52
Distributing deployment to child gears
Activating deployment
Stopping PHP 5.3 cartridge (Apache+mod_php)
Waiting for stop to finish
HAProxy already running
HAProxy instance is started
Starting PHP 5.3 cartridge (Apache+mod_php)

Then all the gears is synced, change take effect now. So this should be RHC client bug.

Comment 2 Andy Goldstein 2014-01-27 14:27:36 UTC

"gear remotedeploy" is only intended to be executed by CI builders. The more appropriate invocation here is "gear activate --all"

Comment 3 Fabiano Franz 2014-01-28 03:40:04 UTC

The rhc command was originally calling 'gear activate'. Fixed by changing to 'gear activate --all' in https://github.com/openshift/rhc/pull/537.

(In reply to Andy Goldstein from comment #2)
> "gear remotedeploy" is only intended to be executed by CI builders. The more
> appropriate invocation here is "gear activate --all"

Comment 4 openshift-github-bot 2014-01-28 04:45:13 UTC

Commit pushed to master at https://github.com/openshift/rhc

https://github.com/openshift/rhc/commit/669697f29d240dd263bef2a4767008206452e18f
Bug 1058251 - gear activate requires --all to be applied to all gears

Comment 5 Meng Bo 2014-01-28 06:04:28 UTC

Checked with rhc client which built from the latest source code on github.
Issue has been fixed.

[bmeng@localhost php54s]$ rhc ssh php54s --gear "gear deployments"
=== 52e7465e7625230aa6000003 php-5.4+haproxy-1.4
Activation time - Deployment ID - Git Ref - Git SHA1
2014-01-28 00:58:42 -0500 - 23204687 - master - 120be63 - ACTIVE
2014-01-28 00:54:23 -0500 - 32371551 - master - fa8029d
[52e7465e7625230aa6000003 php-5.4+haproxy-1.4] 
=== 52e745ef762523b56d00001f php-5.4+haproxy-1.4
Activation time - Deployment ID - Git Ref - Git SHA1
2014-01-28 00:58:35 -0500 - 23204687 - master - 120be63 - ACTIVE
2014-01-28 00:54:23 -0500 - 32371551 - master - fa8029d
[52e745ef762523b56d00001f php-5.4+haproxy-1.4] 

[bmeng@localhost php54s]$ rhc deployment-activate 32371551
Activating deployment '32371551' on application php54s ...
Activating deployment
Stopping PHP 5.4 cartridge (Apache+mod_php)
Waiting for stop to finish
HAProxy already running
HAProxy instance is started
Starting PHP 5.4 cartridge (Apache+mod_php)
Success

[bmeng@localhost php54s]$ rhc ssh php54s --gear "gear deployments"
=== 52e7465e7625230aa6000003 php-5.4+haproxy-1.4
Activation time - Deployment ID - Git Ref - Git SHA1
2014-01-28 01:00:50 -0500 - 32371551 - master - fa8029d - ACTIVE
2014-01-28 00:58:42 -0500 - 23204687 - master - 120be63
[52e7465e7625230aa6000003 php-5.4+haproxy-1.4] 
=== 52e745ef762523b56d00001f php-5.4+haproxy-1.4
Activation time - Deployment ID - Git Ref - Git SHA1
2014-01-28 01:00:38 -0500 - 32371551 - master - fa8029d - ACTIVE
2014-01-28 00:58:35 -0500 - 23204687 - master - 120be63
[52e745ef762523b56d00001f php-5.4+haproxy-1.4] 


Move bug to verified.