Red Hat Bugzilla – Bug 867349
Got error when add cartridges to scalable app after setting its min gears number bigger than its real gear size .
Last modified: 2015-05-14 21:15:05 EDT
Created attachment 628687 [details]
error message when add mysql.
Description of problem:
When I tested US2283 on devenv_2340, I found that after I set the min gear size number bigger than the real app gear size for twice, I would get error when I tried to add cartridges to the app. The error page was attached. The development.log when add cartridge was also attached.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1.Create a scalable app:
$rhc app create -s -a spython22 -t python-2.6 -l e
2.Go to the app detail page, and clicked the "scaled up with HAProxy" link.
3.Choose 8 as the min gear size.
4.Add cartridge mongodb.
5.Choose 12 as the min gear size.
6.Add cartridge mysql.
After step 4, the mongdb can be embed successfully and the gear size of the app would be scaled up to 9.
After step 6. I got error message :"Consistency check failed. Could not update application 'spython252' for 'e'", and cannot added mysql successfully.And the gear size of the app was still 9.
After step 6, the mysql should be embed successfully, and the gear size of the app should be scaled up to 13.
I have tried to use the REST API as follows to scale up the app after step 6, and it would get error messages such as "Consistency check failed." or "Unable to create gear on node".
The REST API to scale up:
curl -k -X POST -H 'Accept: application/xml' -d event=scale-up --user e:e https://ec2-54-242-72-151.compute-1.amazonaws.com/broker/rest/domains/domainnnn/applications/spython252/events
Created attachment 628688 [details]
development.log when add mysql at step 6.
Cannot reproduce this problem. I tried this several times over with each line on the debugger too - no issues.
Now several theoretical issues are possible which can result in that problem and one of them indeed caused the problem that you saw. I took the liberty of starting over your ec2 instance and debugging on it. From the given username/appname I see that the mongo datastore for user 'e' is inconsistent - the consumed_gears is 15 and the current total taken by the app is 6. So here is what went wrong -
1. Set the scales_from to 10
(at this point the consumed_gear count was 15 and max_gears is 16)
2. add mysql to the app
(mysql got a good gear and the consumed_gear count went up to 16)
3. but since scales_from was increased, configuring the dependencies also resulted in scaling up the python gears...
(as soon as the first gear was created, the mongo save failed because consumed_gears == max_gears)... ERROR
Now even if your consumed_gear vs actual gear count were consistent you could have hit this issue - as soon as the max_gears count is hit the said error is raised.
I do not see this bug as a functional issue - the app was still restored alright in correct state. The messaging was wrong - instead of saying 'Consistency check failed', the message should be 'you have reached the max_gears for this account'.
I will put in a fix for correcting the message after I get around some other issues I have discovered on the way.
Fixed with rev#cb8dbc6bbc2014f319239724d035470ffa30ae13 in origin-server/master
(In reply to comment #3)
Tested this issue on devev_2360, it has been fixed now, mongodb and mysql can be added successfully, thanks.