Bug 965028 - One gear is down for a scaling jbosseap-6.0/jbossas-7.0/jbossews application after setting min gear values larger than or equal to 3
One gear is down for a scaling jbosseap-6.0/jbossas-7.0/jbossews application...
Status: CLOSED CURRENTRELEASE
Product: OpenShift Online
Classification: Red Hat
Component: Containers (Show other bugs)
2.x
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Dan Mace
libra bugs
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2013-05-20 06:41 EDT by Wei Sun
Modified: 2015-05-14 19:19 EDT (History)
1 user (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-06-11 00:06:37 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
eaps (128.98 KB, image/png)
2013-05-20 06:43 EDT, Wei Sun
no flags Details
jbossas (132.14 KB, image/png)
2013-05-20 06:43 EDT, Wei Sun
no flags Details
jbossews (103.83 KB, image/png)
2013-05-20 06:44 EDT, Wei Sun
no flags Details
eap-haproxy-status (134.11 KB, image/png)
2013-05-22 02:02 EDT, Wei Sun
no flags Details

  None (edit)
Description Wei Sun 2013-05-20 06:41:43 EDT
Description of problem:
After setting min gear value larger than or equal to 3 for a scaling jboss app,and check /haproxy-status/ page,but one gear is down.And run `rhc app-show $appname --gear`,the down gear is 
shown as "new" state.

Version-Release number of selected component (if applicable):
devenv_3247

How reproducible:
Always

Steps to Reproduce:
1.Create a scaling jbosseap-6.0 app
2.Add disable auto scaling marker
3.Set min gear value as 3
4.Run `rhc app-show $appname gear`
5.Check /haproxy-status/
Actual results:
4.[openshift@localhost ~]$ rhc app-show eaps --gear
ID                               State   Cartridges               Size  SSH URL
-------------------------------- ------- ------------------------ ----- ---------------------------------------------------------------------------------------------
636740365859622779617280         started jbosseap-6.0 haproxy-1.4 small 636740365859622779617280@eaps-wsundev3247.dev.rhcloud.com
06a0b08cc13011e2b6f812313d01406d started jbosseap-6.0 haproxy-1.4 small 06a0b08cc13011e2b6f812313d01406d@06a0b08cc13011e2b6f812313d01406d-wsundev3247.dev.rhcloud.com
5199ed66990e9755ef000002         new jbosseap-6.0 haproxy-1.4 small 5199ed66990e9755ef000002@5199ed66990e9755ef000002-wsundev3247.dev.rhcloud.com
5.One gear is down,please refer to attachment.

Expected results:
All gears are started

Additional info:
Comment 1 Wei Sun 2013-05-20 06:43:29 EDT
Created attachment 750493 [details]
eaps
Comment 2 Wei Sun 2013-05-20 06:43:57 EDT
Created attachment 750494 [details]
jbossas
Comment 3 Wei Sun 2013-05-20 06:44:36 EDT
Created attachment 750496 [details]
jbossews
Comment 4 Dan Mace 2013-05-21 16:00:27 EDT
Should be resolved for this case by https://github.com/openshift/origin-server/pull/2578.

I should also note that these sorts of scaling tests (esp. with jboss cartridges) should be performed in a multi-node environment setup: As the scale-up events are executed in parallel, performing this scenario on a single node will rapidly eat resources and likely cause the operation to take a very long time and hit some timeout. In a clustered environment, the gears will be balanced across other physical nodes.
Comment 5 openshift-github-bot 2013-05-21 18:48:14 EDT
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/4e0393a67acc0ebc1b06578b31097353901761b8
Bug 965028: Increase connector timeout
Comment 6 Wei Sun 2013-05-22 01:54:59 EDT
Check this bug in a multi-node environment and single node,the bug is not fixed.

After setting min gear value as 3,and try to access the app's url,but it is redirected to /haproxy-status/ page,and two gears are down,can not access the down gear.
Please refer to attachment.
Comment 7 Wei Sun 2013-05-22 02:01:28 EDT
(In reply to Wei Sun from comment #6)
> Check this bug in a multi-node environment and single node,the bug is not
> fixed.
> 
> After setting min gear value as 3,and try to access the app's url,but it is
> redirected to /haproxy-status/ page,and two gears are down,can not access
> the down gear.
> Please refer to attachment.

Check this bug on devenv_3257
Comment 8 Wei Sun 2013-05-22 02:02:03 EDT
Created attachment 751506 [details]
eap-haproxy-status
Comment 9 Dan Mace 2013-05-22 16:52:53 EDT
Wei,

You're experiencing a new general scaling bug:

https://bugzilla.redhat.com/show_bug.cgi?id=966068

Which is resolved by:

https://github.com/openshift/origin-server/pull/2600

Please re-test with the related bug's resolution in place.

Thanks.
Comment 10 zhaozhanqi 2013-05-23 06:34:22 EDT
Tested this bug on devenv_3262,it has been fixed

for jbosseap:

[redhat@localhost shell]$ rhc cartridge scale -a zqjbosseap -c jbosseap-6.0 --min 3
RESULT:

jbosseap-6.0 (JBoss Enterprise Application Platform 6.0)
--------------------------------------------------------
  Scaling: x3 (minimum: 3, maximum: available) on small gears
This gear costs an additional $0.03 per gear after the first 3 gears.

Success: Scaling values updated
[redhat@localhost shell]$ rhc app show zqjbosseap -g
ID                               State   Cartridges               Size  SSH URL
-------------------------------- ------- ------------------------ ----- --------------------------------------------------------------------------------------
519dec75d2554c08f6000003         started jbosseap-6.0 haproxy-1.4 small 519dec75d2554c08f6000003@zqjbosseap-zqdd.dev.rhcloud.com
58cdefaac39211e28f3522000a939c30 started jbosseap-6.0 haproxy-1.4 small 58cdefaac39211e28f3522000a939c30@58cdefaac39211e28f3522000a939c30-zqdd.dev.rhcloud.com
391132355727699442925568         started jbosseap-6.0 haproxy-1.4 small 391132355727699442925568@391132355727699442925568-zqdd.dev.rhcloud.com

for ews-2.0

[zqzhao@dhcp-13-222 myshell]$ rhc cartridge scale -a zqjbossews20s -c jbossews-2.0 --min 3
RESULT:

jbossews-2.0 (Tomcat 7 (JBoss EWS 2.0))
---------------------------------------
  Scaling: x3 (minimum: 3, maximum: available) on small gears

Success: Scaling values updated


[zqzhao@dhcp-13-222 myshell]$ rhc app show zqjbossews20s -g
ID                       State   Cartridges               Size  SSH URL
------------------------ ------- ------------------------ ----- ---------------------------------------------------------------------
519d83865033da0a7e000004 started jbossews-2.0 haproxy-1.4 small 519d83865033da0a7e000004@zqjbossews20s-zqd.dev.rhcloud.com
773120243211480156078080 started jbossews-2.0 haproxy-1.4 small 773120243211480156078080@773120243211480156078080-zqd.dev.rhcloud.com
876520480183240169095168 started jbossews-2.0 haproxy-1.4 small 876520480183240169095168@876520480183240169095168-zqd.dev.rhcloud.com

for jbossas-7:

[zqzhao@dhcp-13-222 myshell]$ rhc app show zqjbossas2  -g
ID                               State   Cartridges            Size  SSH URL
-------------------------------- ------- --------------------- ----- -------------------------------------------------------------------------------------
519dedc55033da49f5000013         started jbossas-7 haproxy-1.4 small 519dedc55033da49f5000013@zqjbossas2-zqd.dev.rhcloud.com
0a7072f0c39311e28d8222000a91e2af started jbossas-7 haproxy-1.4 small 0a7072f0c39311e28d8222000a91e2af@0a7072f0c39311e28d8222000a91e2af-zqd.dev.rhcloud.com
807847749717487591096320         started jbossas-7 haproxy-1.4 small 807847749717487591096320@807847749717487591096320-zqd.dev.rhcloud.com

Note You need to log in before you can comment on or make changes to this bug.