Bug 965028

Summary: One gear is down for a scaling jbosseap-6.0/jbossas-7.0/jbossews application after setting min gear values larger than or equal to 3
Product: OpenShift Online Reporter: Wei Sun <wsun>
Component: ContainersAssignee: Dan Mace <dmace>
Status: CLOSED CURRENTRELEASE QA Contact: libra bugs <libra-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 2.xCC: zzhao
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-06-11 04:06:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
eaps
none
jbossas
none
jbossews
none
eap-haproxy-status none

Description Wei Sun 2013-05-20 10:41:43 UTC
Description of problem:
After setting min gear value larger than or equal to 3 for a scaling jboss app,and check /haproxy-status/ page,but one gear is down.And run `rhc app-show $appname --gear`,the down gear is 
shown as "new" state.

Version-Release number of selected component (if applicable):
devenv_3247

How reproducible:
Always

Steps to Reproduce:
1.Create a scaling jbosseap-6.0 app
2.Add disable auto scaling marker
3.Set min gear value as 3
4.Run `rhc app-show $appname gear`
5.Check /haproxy-status/
Actual results:
4.[openshift@localhost ~]$ rhc app-show eaps --gear
ID                               State   Cartridges               Size  SSH URL
-------------------------------- ------- ------------------------ ----- ---------------------------------------------------------------------------------------------
636740365859622779617280         started jbosseap-6.0 haproxy-1.4 small 636740365859622779617280.rhcloud.com
06a0b08cc13011e2b6f812313d01406d started jbosseap-6.0 haproxy-1.4 small 06a0b08cc13011e2b6f812313d01406d.rhcloud.com
5199ed66990e9755ef000002         new jbosseap-6.0 haproxy-1.4 small 5199ed66990e9755ef000002.rhcloud.com
5.One gear is down,please refer to attachment.

Expected results:
All gears are started

Additional info:

Comment 1 Wei Sun 2013-05-20 10:43:29 UTC
Created attachment 750493 [details]
eaps

Comment 2 Wei Sun 2013-05-20 10:43:57 UTC
Created attachment 750494 [details]
jbossas

Comment 3 Wei Sun 2013-05-20 10:44:36 UTC
Created attachment 750496 [details]
jbossews

Comment 4 Dan Mace 2013-05-21 20:00:27 UTC
Should be resolved for this case by https://github.com/openshift/origin-server/pull/2578.

I should also note that these sorts of scaling tests (esp. with jboss cartridges) should be performed in a multi-node environment setup: As the scale-up events are executed in parallel, performing this scenario on a single node will rapidly eat resources and likely cause the operation to take a very long time and hit some timeout. In a clustered environment, the gears will be balanced across other physical nodes.

Comment 5 openshift-github-bot 2013-05-21 22:48:14 UTC
Commit pushed to master at https://github.com/openshift/origin-server

https://github.com/openshift/origin-server/commit/4e0393a67acc0ebc1b06578b31097353901761b8
Bug 965028: Increase connector timeout

Comment 6 Wei Sun 2013-05-22 05:54:59 UTC
Check this bug in a multi-node environment and single node,the bug is not fixed.

After setting min gear value as 3,and try to access the app's url,but it is redirected to /haproxy-status/ page,and two gears are down,can not access the down gear.
Please refer to attachment.

Comment 7 Wei Sun 2013-05-22 06:01:28 UTC
(In reply to Wei Sun from comment #6)
> Check this bug in a multi-node environment and single node,the bug is not
> fixed.
> 
> After setting min gear value as 3,and try to access the app's url,but it is
> redirected to /haproxy-status/ page,and two gears are down,can not access
> the down gear.
> Please refer to attachment.

Check this bug on devenv_3257

Comment 8 Wei Sun 2013-05-22 06:02:03 UTC
Created attachment 751506 [details]
eap-haproxy-status

Comment 9 Dan Mace 2013-05-22 20:52:53 UTC
Wei,

You're experiencing a new general scaling bug:

https://bugzilla.redhat.com/show_bug.cgi?id=966068

Which is resolved by:

https://github.com/openshift/origin-server/pull/2600

Please re-test with the related bug's resolution in place.

Thanks.

Comment 10 zhaozhanqi 2013-05-23 10:34:22 UTC
Tested this bug on devenv_3262,it has been fixed

for jbosseap:

[redhat@localhost shell]$ rhc cartridge scale -a zqjbosseap -c jbosseap-6.0 --min 3
RESULT:

jbosseap-6.0 (JBoss Enterprise Application Platform 6.0)
--------------------------------------------------------
  Scaling: x3 (minimum: 3, maximum: available) on small gears
This gear costs an additional $0.03 per gear after the first 3 gears.

Success: Scaling values updated
[redhat@localhost shell]$ rhc app show zqjbosseap -g
ID                               State   Cartridges               Size  SSH URL
-------------------------------- ------- ------------------------ ----- --------------------------------------------------------------------------------------
519dec75d2554c08f6000003         started jbosseap-6.0 haproxy-1.4 small 519dec75d2554c08f6000003.rhcloud.com
58cdefaac39211e28f3522000a939c30 started jbosseap-6.0 haproxy-1.4 small 58cdefaac39211e28f3522000a939c30.rhcloud.com
391132355727699442925568         started jbosseap-6.0 haproxy-1.4 small 391132355727699442925568.rhcloud.com

for ews-2.0

[zqzhao@dhcp-13-222 myshell]$ rhc cartridge scale -a zqjbossews20s -c jbossews-2.0 --min 3
RESULT:

jbossews-2.0 (Tomcat 7 (JBoss EWS 2.0))
---------------------------------------
  Scaling: x3 (minimum: 3, maximum: available) on small gears

Success: Scaling values updated


[zqzhao@dhcp-13-222 myshell]$ rhc app show zqjbossews20s -g
ID                       State   Cartridges               Size  SSH URL
------------------------ ------- ------------------------ ----- ---------------------------------------------------------------------
519d83865033da0a7e000004 started jbossews-2.0 haproxy-1.4 small 519d83865033da0a7e000004.rhcloud.com
773120243211480156078080 started jbossews-2.0 haproxy-1.4 small 773120243211480156078080.rhcloud.com
876520480183240169095168 started jbossews-2.0 haproxy-1.4 small 876520480183240169095168.rhcloud.com

for jbossas-7:

[zqzhao@dhcp-13-222 myshell]$ rhc app show zqjbossas2  -g
ID                               State   Cartridges            Size  SSH URL
-------------------------------- ------- --------------------- ----- -------------------------------------------------------------------------------------
519dedc55033da49f5000013         started jbossas-7 haproxy-1.4 small 519dedc55033da49f5000013.rhcloud.com
0a7072f0c39311e28d8222000a91e2af started jbossas-7 haproxy-1.4 small 0a7072f0c39311e28d8222000a91e2af.rhcloud.com
807847749717487591096320         started jbossas-7 haproxy-1.4 small 807847749717487591096320.rhcloud.com