Red Hat Bugzilla – Bug 871660
Scaled application has "dud" gears when overlapping scale-up requests occur
Last modified: 2015-05-14 22:07:37 EDT
Created attachment 635887 [details]
Double configure error logs
Description of problem:
I was running Apache benchmark on a scaled application and noticed that there was a "dud" gear in the haproxy configuration. On investigating this, found that configure was called twice on a gear and that gear's dns was never removed.
Version-Release number of selected component (if applicable):
Current release (2.0.19)
Steps to Reproduce:
1. Create a scaled application
2. Add mysql to the app
3. Use the rails quickstart - https://github.com/openshift/rails-example
4. Add a junk folder with 2K files (201 MB worth) -- this is to simulate a customer
issue on prod where there were a lot of files and sync was taking a while.
for i in `seq 2048`; do dd if=/dev/zero of=junk-$i count=100 bs=1024; done
5. Add, commit and push to the app.
6. On the devenv, run apache benchmark
ab -n 100000 -c 23 http://$app-$namespace.dev.rhcloud.com/
7. You may optionally manually scale up while a scale up event occurs to
reproduce this (seems to happen when scaleups occur in parallel)
Should expect a valid configuration and haproxy running correctly
Mismatch between haproxy configs (running + gear registry).
This also leaves a lot of defunct shell process -- run from mcollective. The reason
is the sync gears takes a while because of the # of files + size and because ab is actually "bashing" the haproxy gear with requests.
At the very least we should clear up the gear's dns entry -- we do delete the gear.
In the attached logs -- look for the gear b1db43744f6a4fd7934c165ba53373c0 and you'll see configure called twice:
at line #s 10391 and 10802 in the mcollective logs:
10393 :args=>"'b1db43744f' 'rr50' 'b1db43744f6a4fd7934c165ba53373c0'",
10804 :args=>"'b1db43744f' 'rr50' 'b1db43744f6a4fd7934c165ba53373c0'",
on the same gear.
Should be fixed with model refactor
*** This bug has been marked as a duplicate of bug 855307 ***