Bug 851494 - Gears will be down if disable auto scaling and hot deploy are both triggered
Gears will be down if disable auto scaling and hot deploy are both triggered
Status: CLOSED CURRENTRELEASE
Product: OpenShift Origin
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
low Severity low
: ---
: ---
Assigned To: Ram Ranganathan
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2012-08-24 06:43 EDT by joycezhang
Modified: 2015-05-14 18:58 EDT (History)
4 users (show)

See Also:
Fixed In Version: devenv_2196
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2012-11-06 13:48:12 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
scale event log (3.87 KB, text/x-log)
2012-08-24 06:43 EDT, joycezhang
no flags Details

  None (edit)
Description joycezhang 2012-08-24 06:43:01 EDT
Created attachment 606827 [details]
scale event log

Description of problem:
The function of disable auto scaling is not stable enough as the gears are down sometimes, this problem is happened especially after scaling up at the first time of the app.

Version-Release number of selected component (if applicable):
devenv-stage_234
rhc-0.97.17.gem

How reproducible:
Not always

Steps to Reproduce:
1. Create a scanable appp
#rhc app create -a ruby18s9 -t ruby-1.8 -p xxx
2.Disable auto scaling by:

create new file $APP_Dir/.openshift/markers/disable_auto_scaling

cd $APP

touch .openshift/markers/disable_auto_scaling

git commit -a -m "disable_auto_scaling"

git push

3. Use Rest API to scale up the gears
curl -k -H "Accept: application/xml" --user "jinzhang+6@redhat.com:redhat" https://stg.openshift.redhat.com/broker/rest/domains/stgdm6/applications/ruby18s9/events -d event=scale-up

4. Scale up again:(The gears are not up after step3 especially at the first time)
curl -k -H "Accept: application/xml" --user "jinzhang+6@redhat.com:redhat" https://stg.openshift.redhat.com/broker/rest/domains/stgdm6/applications/ruby18s9/events -d event=scale-up

5. Wait for a few minites , then check the gears status.
  
Actual results:
The gears are down.



Expected results:
All gears are running as the app has already disabled auto scaling.


Additional info:
This bug is reproduced after scaling up at the first time when the auto scaling disabled.
If try to scale up the third time, the gears will be running correctly.

Please refer to the log as attached.
Comment 1 joycezhang 2012-08-24 06:57:34 EDT
Some additional info need to be updated here for the reference:

If do the git push with auto scaling disabled at the first time, the gears will be down always even if using rest api to scale up again and again.(The git push is completed successfully in this situation)

Then if try to git push with auto scaling disabled again(do the same steps as described in step2 above), then the gears will be running persistently.
Comment 2 joycezhang 2012-08-27 01:32:37 EDT
This bug is also reproduced on devenv_2078.
Comment 3 joycezhang 2012-08-29 05:44:00 EDT
This bug can be reproduced on devenv_2091 especially has hot_deploy marker triggerred.
Comment 4 joycezhang 2012-08-29 05:54:51 EDT
Please refer to the updated reproduce steps as below:

Steps to Reproduce:
1. Create a scanable appp
#rhc app create -a ruby18s9 -t ruby-1.8 -p xxx
2.Disable auto scaling by:

create new file $APP_Dir/.openshift/markers/disable_auto_scaling

cd $APP

touch .openshift/markers/disable_auto_scaling
3. Add hot deploy marker
touch .openshift/markers/hot_deploy

4.git push

5. Use Rest API to scale up the gears
curl -k -H "Accept: application/xml" --user "jinzhang+6@redhat.com:redhat" https://stg.openshift.redhat.com/broker/rest/domains/stgdm6/applications/ruby18s9/events -d event=scale-up

6. Wait for a few minites , then check the gears status.

Actual resluts:
Gears will be down after a few minutes.
Comment 5 Ram Ranganathan 2012-09-05 01:32:46 EDT
Marking this as low (or guess it can even be closed as it works as designed) -- the workaround is to first disable auto scaling, do a git add+commit+push and then do the hot_deploy marker. 

If you add both of those as part of 1git push, then hot deploy takes precedence and does not stop the server -- which means the haproxy_ctld will continue to run and auto scale up/down the newly add gears. 

So remove the hot_deploy marker and things will function as you expect.
And then turn on hot deployment.
Comment 6 Ram Ranganathan 2012-09-18 21:26:38 EDT
Fixed and waiting on pull request: 
   https://github.com/openshift/crankcase/pull/504
Comment 7 Xiaoli Tian 2012-09-19 00:50:16 EDT
(In reply to comment #6)
> Fixed and waiting on pull request: 
>    https://github.com/openshift/crankcase/pull/504

Above pull is merged in devenv_2196. move it to ON_QA to verify.
Comment 8 joycezhang 2012-09-19 04:23:06 EDT
This bug has been verified and fixed. 

Pls get more details as below:

Verified build:
devenv_2196
rhc-0.99.1.gem

Verified steps:
1. Create a scalable app
2. Add disable auto scaling marker
3. Add hot deploy marker
4. Git push to trigger the two markders added above.
5. Scale up by rest api


Verified results:
The gears are running as exprected. 

Here's the scale events log for your reference:
[ruby18sy-joycdev2.dev.rhcloud.com logs]\> cat scale_events.log 
# Logfile created on Wed Sep 19 03:50:56 -0400 2012 by logger.rb/1.2.6
I, [2012-09-19T03:51:36.458568 #27880]  INFO -- : Starting haproxy_ctld
D, [2012-09-19T03:51:36.460352 #27880] DEBUG -- : GEAR_INFO - capacity: 0.0% gear_count: 1 sessions: 0 up/remove_thresh: 90.0%/49.9% sec_left_til_remove: 120 gear_remove_thresh: 0/20
I, [2012-09-19T04:16:10.401776 #8404]  INFO -- : Starting haproxy_ctld
D, [2012-09-19T04:16:10.403774 #8404] DEBUG -- : GEAR_INFO - capacity: 0.0% gear_count: 2 sessions: 0 up/remove_thresh: 90.0%/49.9% sec_left_til_remove: 120 gear_remove_thresh: 0/20

Mark it as Verified after verification. Thanks.

Note You need to log in before you can comment on or make changes to this bug.