Created attachment 606827 [details] scale event log Description of problem: The function of disable auto scaling is not stable enough as the gears are down sometimes, this problem is happened especially after scaling up at the first time of the app. Version-Release number of selected component (if applicable): devenv-stage_234 rhc-0.97.17.gem How reproducible: Not always Steps to Reproduce: 1. Create a scanable appp #rhc app create -a ruby18s9 -t ruby-1.8 -p xxx 2.Disable auto scaling by: create new file $APP_Dir/.openshift/markers/disable_auto_scaling cd $APP touch .openshift/markers/disable_auto_scaling git commit -a -m "disable_auto_scaling" git push 3. Use Rest API to scale up the gears curl -k -H "Accept: application/xml" --user "jinzhang+6:redhat" https://stg.openshift.redhat.com/broker/rest/domains/stgdm6/applications/ruby18s9/events -d event=scale-up 4. Scale up again:(The gears are not up after step3 especially at the first time) curl -k -H "Accept: application/xml" --user "jinzhang+6:redhat" https://stg.openshift.redhat.com/broker/rest/domains/stgdm6/applications/ruby18s9/events -d event=scale-up 5. Wait for a few minites , then check the gears status. Actual results: The gears are down. Expected results: All gears are running as the app has already disabled auto scaling. Additional info: This bug is reproduced after scaling up at the first time when the auto scaling disabled. If try to scale up the third time, the gears will be running correctly. Please refer to the log as attached.
Some additional info need to be updated here for the reference: If do the git push with auto scaling disabled at the first time, the gears will be down always even if using rest api to scale up again and again.(The git push is completed successfully in this situation) Then if try to git push with auto scaling disabled again(do the same steps as described in step2 above), then the gears will be running persistently.
This bug is also reproduced on devenv_2078.
This bug can be reproduced on devenv_2091 especially has hot_deploy marker triggerred.
Please refer to the updated reproduce steps as below: Steps to Reproduce: 1. Create a scanable appp #rhc app create -a ruby18s9 -t ruby-1.8 -p xxx 2.Disable auto scaling by: create new file $APP_Dir/.openshift/markers/disable_auto_scaling cd $APP touch .openshift/markers/disable_auto_scaling 3. Add hot deploy marker touch .openshift/markers/hot_deploy 4.git push 5. Use Rest API to scale up the gears curl -k -H "Accept: application/xml" --user "jinzhang+6:redhat" https://stg.openshift.redhat.com/broker/rest/domains/stgdm6/applications/ruby18s9/events -d event=scale-up 6. Wait for a few minites , then check the gears status. Actual resluts: Gears will be down after a few minutes.
Marking this as low (or guess it can even be closed as it works as designed) -- the workaround is to first disable auto scaling, do a git add+commit+push and then do the hot_deploy marker. If you add both of those as part of 1git push, then hot deploy takes precedence and does not stop the server -- which means the haproxy_ctld will continue to run and auto scale up/down the newly add gears. So remove the hot_deploy marker and things will function as you expect. And then turn on hot deployment.
Fixed and waiting on pull request: https://github.com/openshift/crankcase/pull/504
(In reply to comment #6) > Fixed and waiting on pull request: > https://github.com/openshift/crankcase/pull/504 Above pull is merged in devenv_2196. move it to ON_QA to verify.
This bug has been verified and fixed. Pls get more details as below: Verified build: devenv_2196 rhc-0.99.1.gem Verified steps: 1. Create a scalable app 2. Add disable auto scaling marker 3. Add hot deploy marker 4. Git push to trigger the two markders added above. 5. Scale up by rest api Verified results: The gears are running as exprected. Here's the scale events log for your reference: [ruby18sy-joycdev2.dev.rhcloud.com logs]\> cat scale_events.log # Logfile created on Wed Sep 19 03:50:56 -0400 2012 by logger.rb/1.2.6 I, [2012-09-19T03:51:36.458568 #27880] INFO -- : Starting haproxy_ctld D, [2012-09-19T03:51:36.460352 #27880] DEBUG -- : GEAR_INFO - capacity: 0.0% gear_count: 1 sessions: 0 up/remove_thresh: 90.0%/49.9% sec_left_til_remove: 120 gear_remove_thresh: 0/20 I, [2012-09-19T04:16:10.401776 #8404] INFO -- : Starting haproxy_ctld D, [2012-09-19T04:16:10.403774 #8404] DEBUG -- : GEAR_INFO - capacity: 0.0% gear_count: 2 sessions: 0 up/remove_thresh: 90.0%/49.9% sec_left_til_remove: 120 gear_remove_thresh: 0/20 Mark it as Verified after verification. Thanks.