Red Hat Bugzilla – Bug 74543
Using kernel semaphores is problematic for hosting and makes denial of service attack easy.
Last modified: 2014-01-21 17:48:03 EST
From Bugzilla Helper: User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0.0) Gecko/20020606 Description of problem: The linux kernel only has 128 semaphore arrays total. Each apache running uses up 4 of those. This makes running separate httpd's for each customer on a hosting box problematic and hosting more than 32 sites this way impossible. This is WAY below the capabilities of most hosting hardware. Additionally, since any user can now start an apache instance, this makes a DOS attack trivial. The old version of apache+mm used files instead, which were written to /var/run. I believe "null" does the right thing by reverting back to this behaviour, and putting the semaphore files under ServerRoot. Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. Run apache httpd as root 2. Check number of semaphore arrays in use using ipcs -s 3. Run additional httpds as non-root user 4. Check number of semaphore arrays in use using ipcs -s Actual Results: Semaphore arrays will fill up to the maximum 128 Starting additiona apache httpd will fail because there are no more semaphores available. Additional info: See also bug https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=70846 for additional information + discussion.
Howdy, I was going to submit a bug report for this as well. We've seen a number of customers having problems with high traffic web apps on 7.X getting stuff like: Starting httpd: Ouch! ap_mm_create(1048576, "/var/run/httpd.mm.20330") failed Error: MM: mm:core: failed to acquire semaphore (No space left on device): OS: Identifier removed Would it be possible to do something like setup the mm spec file to create an RPM for ipcsem compiled mm and another one for fcntl and then use something like alternatives to let the user decide which one to use? In the case of 7.2, I think that fcntl should be used as the default since that's what the distro shipped with.
You can increase the number of semaphores by setting a parameter in /proc. Under RedHat, add the following to /etc/sysctl.conf kernel.sem = 250 32000 100 256 The last number is the number of semaphore arrays (default is 128). After editing /etc/sysctl.conf run sysctl -p
This is the same bug as described in #72158. I just ran across it today for the first, triggered apparently by a large number of apache restarts yesterday while I was bringing a new config up. Using sysctl to increase the maximum number of semaphores temporarily solves the problem, at least for however long it takes for the number of semaphores to reach that maximum again. Bug #72158 was opened in late August, this bug was opened in late September and I didn't see any Red Hat response to either yet - both are still set at NEW. Is there going to be any real errata/patches to fix this problem or are we just sitting out there with a known bug but no hope for support (except maybe upgrading to Apache 2.X which does away with mm but has other problems, like PHP compatibility/stability)... Just curious - it doesn't seem to be an isolated incident (affects at least all 7.X Red Hat releases), its reproducible, and the silence about a real fix is deafening. Maybe I'll have to look at the 1.1.3-4 -> 1.1.3-8 diff and see if there's anything obvious - anyone have any bright ideas?
As noted in the initial description there is further information under Bug 70846. In that bug someone from RedHat requested a separate bug report specific to mm. Well this is a separate bug report but appears to be receiving no attention from anyone at RedHat?! Also as this is a DOS issue can whoever has control of such things please elevate this to High priority and Security severity in the hope that someone at RedHat actually takes some notice of this.
This is a huge problem. The SysV IPC objects have always been a complete disaster. I've opted to recompile and reload the mm library. ./configure --with-shm=MMANON --with-sem=FLOCK Would be nice if you could impose the order of search in ldconfig so you could override the system settings by loading things into /usr/local/lib without having to disturb existing packages. I still get one semaphore from httpd.
*** Bug 78922 has been marked as a duplicate of this bug. ***
http://people.redhat.com/jorton/7.x-mm/ for unsupported RPMs; errata currently being prepared.
An errata has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on the solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHBA-2002-273.html
*** Bug 78019 has been marked as a duplicate of this bug. ***
*** Bug 72158 has been marked as a duplicate of this bug. ***
*** Bug 78134 has been marked as a duplicate of this bug. ***
*** Bug 74671 has been marked as a duplicate of this bug. ***
*** Bug 78360 has been marked as a duplicate of this bug. ***
*** Bug 110744 has been marked as a duplicate of this bug. ***