Description of problem: I've created a scaled JBoss EAP application, loaded it up using Blitz.io and tested out adding gears. After the load has started up, I add more gears to my application. However, those gears do not show up in HAProxy (either via the status or configuration) until after a significant delay. Version-Release number of selected component (if applicable): Online as of 3/15/2013 How reproducible: Consistent Steps to Reproduce: 1. Create a scaled JBoss EAP application named scale: rhc app-create scale jbosseap -s 2. Disable auto-scaling cd scale/.openshift/markers touch disable_auto_scaling git add . git commit -m "Disabling auto-scaling" git push 3. Limit the scaling to a single gear rhc cartridge-scale jbosseap --app scale --min 1 --max 1 4. Spin up a load test in Blitz.io. I used the settings '-p 50-500:300 --timeout 5000 http://scale-YOUR_DOMAIN.rhcloud.com/' 5. Wait 45 seconds and then add 3 more gears rhc cartridge-scale jbosseap --app scale --min 1 --max 1 6. Wait until all the gears have started and verify they are directly accessible rhc app-show scale --gears (Also verify in the web console that the gears are STARTED) 7. Check the HAProxy status page for the application - http://scale-YOUR_DOMAIN.rhcloud.com/haproxy-status Actual results: For up to minutes after the gears are created and ready, HAProxy configuration still has a single gear. Expected results: Once the gears are created and accessible, HAProxy adds those gears to the routing configuration. Additional info: Screencast displaying the behavior: http://file.rdu.redhat.com/~mhicks/delayed_haproxy.webm You'll notice that my gears were shown as STARTED at 3:21. However, HAProxy didn't show those gears in the configuration until the test had been aborted and several minutes passed at 7:10.
Sorry, line 5 should read 5. Wait 45 seconds and then add 3 more gears rhc cartridge-scale jbosseap --app scale --min 4 --max 4
This has changed drastically between v1 and v2. For v1 the flow was: Create new gears and they were started Publish to the new gears with the deployment artifacts from the head gear Restart the gears with the new code Add gears to haproxy config I think this restarting was causing your complaints. New v2 logic: 1) Create new gears but do not start them 2) Add gears to haproxy config and haproxy views them as down 3) Publish to the new gears with the deployment artifacts from the head gear 4) Start the new gears 5) haproxy sees the gears as available and starts routing to them It's not blazing fast still with eap but I think this resolves your issues. I would probably argue 2) should probably be moved to the end but don't think it's breaking anything.
Tested it on devenv_3238 1. Create a scale jbosseap app 2. Disable auto-scaling 3. Limit the scaling to a single gear rhc cartridge-scale scalejbosseap3 -c jbosseap-6.0 --min 1 --max 1 4. Send concurrent request ab -n 30000 -c 6 https://scalejbosseap3-domx1.dev.rhcloud.com 5.Wait 45 seconds and then add 2 more gears [root@ip-10-202-27-91 markers]# rhc cartridge-scale -a scalejbosseap3 -c jbosseap-6.0 --min 3 --max 3 RESULT: jbosseap-6.0 (JBoss Enterprise Application Platform 6.0) -------------------------------------------------------- Scaling: x3 (minimum: 3, maximum: 3) on small gears This gear costs an additional $0.03 per gear after the first 3 gears. Success: Scaling values updated 6. Check gears status and check haproxy-status page # rhc app show --gears -a scalejbosseap3 ID State Cartridges Size SSH URL -------------------------------- ------- ------------------------ ----- --------------------------------------------------------------------------------------- 519624d4a33f822eaf000001 started jbosseap-6.0 haproxy-1.4 small 519624d4a33f822eaf000001.rhcloud.com ab445cd4bef011e296af12313b1218ad started jbosseap-6.0 haproxy-1.4 small ab445cd4bef011e296af12313b1218ad.rhcloud.com ab85b120bef011e296af12313b1218ad new jbosseap-6.0 haproxy-1.4 small ab85b120bef011e296af12313b1218ad.rhcloud.com According to above, 2 gears are in started status, In haproxy-status page, all the gears are shown as well, only the local gear and ab445cd4bef011e296af12313b1218ad are in started status, gear ab85b120bef011e296af12313b1218ad is in down status, it seems the original issue in the bug does not happen
According to comment 3, the original issue does not happen again, the started gear could be shown in haproxy-status page timely Filed a new bug 965028 to track the issue about the 3rd scaled up gear is down.