Bug 963490 - Scaling updates in HAProxy are delayed
Scaling updates in HAProxy are delayed
Product: OpenShift Online
Classification: Red Hat
Component: Containers (Show other bugs)
Unspecified Unspecified
unspecified Severity high
: ---
: ---
Assigned To: Mrunal Patel
libra bugs
Depends On:
Blocks: 968994 969007
  Show dependency treegraph
Reported: 2013-05-15 18:02 EDT by Matt Hicks
Modified: 2015-05-14 19:18 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 968994 (view as bug list)
Last Closed: 2013-06-11 00:03:57 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Matt Hicks 2013-05-15 18:02:38 EDT
Description of problem:
I've created a scaled JBoss EAP application, loaded it up using Blitz.io and tested out adding gears.  After the load has started up, I add more gears to my application.  However, those gears do not show up in HAProxy (either via the status or configuration) until after a significant delay.

Version-Release number of selected component (if applicable):
Online as of 3/15/2013

How reproducible:

Steps to Reproduce:
1. Create a scaled JBoss EAP application named scale:
  rhc app-create scale jbosseap -s

2. Disable auto-scaling
  cd scale/.openshift/markers
  touch disable_auto_scaling
  git add .
  git commit -m "Disabling auto-scaling"
  git push

3. Limit the scaling to a single gear
  rhc cartridge-scale jbosseap --app scale --min 1 --max 1

4. Spin up a load test in Blitz.io.  I used the settings
  '-p 50-500:300 --timeout 5000 http://scale-YOUR_DOMAIN.rhcloud.com/'
5. Wait 45 seconds and then add 3 more gears
  rhc cartridge-scale jbosseap --app scale --min 1 --max 1

6. Wait until all the gears have started and verify they are directly accessible
  rhc app-show scale --gears
  (Also verify in the web console that the gears are STARTED)

7. Check the HAProxy status page for the application - http://scale-YOUR_DOMAIN.rhcloud.com/haproxy-status

Actual results:
For up to minutes after the gears are created and ready, HAProxy configuration still has a single gear.

Expected results:
Once the gears are created and accessible, HAProxy adds those gears to the routing configuration.

Additional info:
Screencast displaying the behavior: http://file.rdu.redhat.com/~mhicks/delayed_haproxy.webm

You'll notice that my gears were shown as STARTED at 3:21.  However, HAProxy didn't show those gears in the configuration until the test had been aborted and several minutes passed at 7:10.
Comment 1 Matt Hicks 2013-05-15 18:12:27 EDT
Sorry, line 5 should read

5. Wait 45 seconds and then add 3 more gears
  rhc cartridge-scale jbosseap --app scale --min 4 --max 4
Comment 2 Dan McPherson 2013-05-15 18:42:48 EDT
This has changed drastically between v1 and v2.  For v1 the flow was:

Create new gears and they were started
Publish to the new gears with the deployment artifacts from the head gear
Restart the gears with the new code
Add gears to haproxy config

I think this restarting was causing your complaints.

New v2 logic:
1) Create new gears but do not start them
2) Add gears to haproxy config and haproxy views them as down
3) Publish to the new gears with the deployment artifacts from the head gear
4) Start the new gears
5) haproxy sees the gears as available and starts routing to them

It's not blazing fast still with eap but I think this resolves your issues.  I would probably argue 2) should probably be moved to the end but don't think it's breaking anything.
Comment 3 Xiaoli Tian 2013-05-17 09:10:48 EDT
Tested it on devenv_3238

1. Create a scale jbosseap app

2.  Disable auto-scaling

3. Limit the scaling to a single gear
 rhc cartridge-scale scalejbosseap3 -c jbosseap-6.0 --min 1 --max 1
4. Send concurrent request
 ab -n 30000 -c 6 https://scalejbosseap3-domx1.dev.rhcloud.com

5.Wait 45 seconds and then add 2 more gears
[root@ip-10-202-27-91 markers]# rhc cartridge-scale -a scalejbosseap3 -c jbosseap-6.0 --min 3 --max 3

jbosseap-6.0 (JBoss Enterprise Application Platform 6.0)
  Scaling: x3 (minimum: 3, maximum: 3) on small gears
This gear costs an additional $0.03 per gear after the first 3 gears.

Success: Scaling values updated

6. Check gears status and check haproxy-status page

# rhc app show --gears -a scalejbosseap3
ID                               State   Cartridges               Size  SSH URL
-------------------------------- ------- ------------------------ ----- ---------------------------------------------------------------------------------------
519624d4a33f822eaf000001         started jbosseap-6.0 haproxy-1.4 small 519624d4a33f822eaf000001@scalejbosseap3-domx1.dev.rhcloud.com
ab445cd4bef011e296af12313b1218ad started jbosseap-6.0 haproxy-1.4 small ab445cd4bef011e296af12313b1218ad@ab445cd4bef011e296af12313b1218ad-domx1.dev.rhcloud.com
ab85b120bef011e296af12313b1218ad new jbosseap-6.0 haproxy-1.4 small ab85b120bef011e296af12313b1218ad@ab85b120bef011e296af12313b1218ad-domx1.dev.rhcloud.com

According to above, 2 gears are in started status, 

In haproxy-status page, all the gears are shown as well, only the local gear and ab445cd4bef011e296af12313b1218ad are in started status, gear ab85b120bef011e296af12313b1218ad is in down status,  it seems the original issue  in the bug does not happen
Comment 4 Xiaoli Tian 2013-05-20 06:43:51 EDT
According to comment 3, the original issue does not happen again, the started gear could be shown in haproxy-status page timely

Filed a new bug  965028 to  track the issue about the 3rd scaled up gear is down.

Note You need to log in before you can comment on or make changes to this bug.