Red Hat Bugzilla – Bug 994993
Already at the maximum number of gears
Last modified: 2017-03-08 12:35 EST
Description of problem:
When trying to scale an app I am receiving an "Already at max number of gears allowed for either the app or your account" error. I have the app at -1 max, account is set to 100 gears max. Tried with multiple apps, new and older ones. Verifyied account is set to 100 max with oo-admin-chk-user.
Version-Release number of selected component (if applicable): 1.2.0
How reproducible: Reproduces with any app under my account. Have not tried with new account (using ldap authentication so cannot create a new account for myself)
Steps to Reproduce:
1. Have app attempt to auto scale - Receive error in scale_log
2. Tried to manually set to more than 1 gear in web console. Received "/bin/sh: /var/lib/openshift/520260403443a9fd3a00005f/jbosseap/bin/setup: Permission denied.
Actual results: Received Already at max number of gears allowed for either the app or your account" or "/bin/sh: /var/lib/openshift/520260403443a9fd3a00005f/jbosseap/bin/setup: Permission denied errors. If I watch the /var/lib/openshift directory on my node I can see the new gear get created and then immediately deleted.
Expected results: Should create new gear and start that gear.
Have you run oo-diagnostics on your broker and node hosts? If not, please do so and report any output.
Some background: When you use the Web console to scale the app up, the console sends a REST request to the broker API with your scale-up command, the broker passes that command on to the node, and then on the node the haproxy cartridge's control script sends a REST call to the broker API to initiate the scale-up.
From the error message you are seeing, it looks like when the node runtime sends the REST call to the broker to scale up, the broker responds with 422 Unprocessable Entity, which the node runtime interprets as your hitting the limit on gears:
It will help to diagnose the issue if you can provide some log files:
On the broker host,
(so we can see what REST calls the console is making to the broker and what HTTP codes the broker is sending back)
(so we can look for hints as to why the broker is sending an error response to the node host; maybe we'll luck out and get a backtrace)
On the node host,
(so we can look for any hints in case the node runtime is confused)
Would you mind attaching these log files to this bug report?
Created attachment 784543 [details]
Created attachment 784544 [details]
Created attachment 784545 [details]
Created attachment 784546 [details]
I have uploaded the requested log files.
Running oo-diagnostics only reports warnings for selinux being in permissive mode.
I'm betting this is related to JBoss timing out.
As a sanity test, could you verify you can create other types of scaled gears. You could try something lightweight like a scaled php application. If that's the case we can try working through what could be causing the problem with JBoss on your system.
I have/had a php application that I also attempted to scale and I got the same issues. I was able to scale previously but after a few demo's of the Scaleapp it is no longer working. I also tried removing and recreating the app and still got the same issue.
I just had a co-worker attempt to create a php app with scaling and it gave him the permission denied app. I then tried and was able to create a new php app but was not able to modify the minimum to scale it without getting the permission denied.
I had him retry and it failed again, tried without haproxy and it worked. Tried a new one with haproxy again and it now worked. Tried manually scaling it inside the webconsole and that failed.
Hope this helps.
Can you tail the production.log while triggering the error? There a number of stack traces in the logs so we could be dealing with multiple problems. I'd like to focus on one at a time.
Sorry for the delayed response. In a desperate attempt to get everything working for a presentation on friday I rebooted the broker and nodes. Everything appears to be working now, I was able to do my scaling presentation without getting these errors.
Interesting. There's definitely still something going wrong. Let's leave this bug open for now. Please update us if the problem persists.
I'm going to close this bug for now since we aren't able to reproduce the exact problem as originally reported.
The current plan is to spend time improving the haproxy's interpretation of the Broker's response on scaling events. We believe that was what led to the unhelpful error message.