Red Hat Bugzilla – Bug 72158
Apache start error
Last modified: 2007-04-18 12:45:46 EDT
From Bugzilla Helper: User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Q312461; .NET CLR 1.0.3705) Description of problem: Apache will not start... after typing service httpd start (or restart) I get... Starting httpd: Ouch! ap_mm_create(1048576, "/var/run/httpd.mm.29690") failed Error: MM: mm:core: failed to acquire semaphore (No space left on device): OS: Identifier removed df -h reveals adequete space on all devices. I think the problem may be that apache is not properly releasing semaphores when stopping. ipcs -u... [root@sandbox2 root]# ipcs -u ------ Shared Memory Status -------- segments allocated 4 pages allocated 16653 pages resident 15 pages swapped 0 Swap performance: 0 attempts 0 successes ------ Semaphore Status -------- used arrays = 48 allocated semaphores = 48 ------ Messages: Status -------- allocated queues = 0 used headers = 0 used space = 0 bytes service httpd restart [OK][OK] (paraphrased)... [root@sandbox2 root]# ipcs -u ------ Shared Memory Status -------- segments allocated 4 pages allocated 16653 pages resident 15 pages swapped 0 Swap performance: 0 attempts 0 successes ------ Semaphore Status -------- used arrays = 50 allocated semaphores = 50 ------ Messages: Status -------- allocated queues = 0 used headers = 0 used space = 0 bytes Notice Used Arrays and Allocated Semaphores. They will go up by 2 until there ain't no more. ipcs -s shows only 2 owned by apache at any given time. None if apache is stopped. I'll attach the exact results later (ipcs -s) if need be. service httpd graceful and service httpd reload don't seem to have the same problem. Graceful and mod_perl don't seem to play nice, so that's not really an option. I can ipcrm some semaphores, but as the ownership seems to go to root it's hard to tell which ones to make go away. Version-Release number of selected component (if applicable): apache-1.3.23-14 kernel-smp-2.4.18-5 mm-1.1.3-8 mod_perl-1.26-5 mod_ssl-2.8.7-6 php-4.1.2-7.3.3 php-mysql-4.1.2-7.3.3 php-devel-4.1.2-7.3.3 php-ldap-4.1.2-7.3.3 php-imap-4.1.2-7.3.3 Let me know if any other version information is necessary. How reproducible: Didn't try Steps to Reproduce: 1.I could restart apache over and over until the semaphores run out - that's bad. 2. ipcs -u semaphores used = N 3. service httpd restart 4. ipcs -u semaphores used = N+2 Actual Results: 1. ipcs -u semaphores used = N 2. service httpd restart 3. ipcs -u semaphores used = N+2 Expected Results: 1. ipcs -u semaphores used = N 2. service httpd restart 3. ipcs -u semaphores used = N Additional info: Seems to be some chat about this on the net, but nobody's getting any answers. Maybe mm upgrade related - hadn't noticed before then? apache-1.3.23-14 kernel-smp-2.4.18-5 mm-1.1.3-8 mod_perl-1.26-5 mod_ssl-2.8.7-6 php-4.1.2-7.3.3 php-mysql-4.1.2-7.3.3 php-devel-4.1.2-7.3.3 php-ldap-4.1.2-7.3.3 php-imap-4.1.2-7.3.3 Let me know if any other version information is necessary.
Forgot to mention a very important detail. Rebooting fixes the problem for several days, and there is a custom log rotation script that runs nightly and condrestarts apache. I'll check it out next time things are broken to get the semaphore count.
I have exactly the same problem, on Red Hat 7.2. We recently did an up2date which included the new mm package; this is probably related. Rebooting the machine did fix the problem, although it hasn't been running long enough to see if the problem shows up again due to log rotations.
I just ran into this problem today during a server migration. It's a shared virutal environment, so I was setting up separate instances of httpd to run as the shared UID/GID of the site instead of apache/apache and remove the need to do suexec calls (and be able to run mod_php safely). Everything was working fine as I added and stared up sites until this happened, then everything died. I hadn't found this bug report yet, so I rebooted and all was well again. Since I'm running my httpd processes as the individual users, I may have more luck using identifying semaphores and using ipcrm to clean them, but I'm looking forward to a fix on this. Perhaps a temporary fix would be recompiling apache to not use mm? What would be the impact of recompiling without EAPI_MM_CORE_PATH and EAPI_MM defined? Would this cause more problems or poor performance? While this doesn't fix the problem, you can change your logrotation script so that it doesn't need to HUP/USR1 Apache during logrotation by using "copytruncate" in your logrotation script. Here's a copy of my /etc/logrotate.d/apache script: ### /var/log/httpd/*log { daily rotate 4 copytruncate compress } ###
Just caught this bug today too in a server I'm readying for productin use, an Athlon MP 1800 machine (single CPU on a Tyan Thunder K7 board) running RH 7.3. I've been doing a lot of web server restarts because i'm developing some custom mod_perl modules and I finally had to reboot beacause apache just wouldn't start. This needs fixed ASAP!
I have apache packages (1.3.26) and php packages (4.1.2) compiled without mm support. I'll let everyone know how it performs. I'm still playing with RH 7.2, so my compiles that platform, but they should install on 7.3 just fine. If anyone wants my SRPMS or RPMS, just drop me an email.
Forgot to mention a recompiled version of mod_ssl minus MM (2.8.9).
We have the same problem after we've applied the mm-1.1.3-8 fix. This happens to several machines where httpd will be restarted often. The problem will disappear after reboot.
I ended up dumping my multi virt/httpd config and just started using safe_mod and open_basedir directives. At any rate, my non-MM compiled versions of Apache+mod_ssl+php use 1 semaphore (for the parent process), but none of the children use any semaphores. I'm going to run some tests on another server, restarting httpd many times to see if it leaks semaphores. I'll let you know the results later next week unless a fix is found before then.
Just ran across the same problem after installing the mm-1.1.3-8 update on a Redhat 7.3 system. The used arrays and allocated semaphores go up by two every time apache is restarted. As this occurs on a development box, apache can be restarted fairly frequently. Any further ideas on the problem or a fix yet?
mm-1.1.3-4 will "solve" the problem. MM from source is currently at 1.2.1 - http://www.ossp.org/pkg/lib/mm/. -Bill
You can use your apache server after use this command line. ipcs -a|grep apache|awk -F\ '{ print }' |xargs ipcrm sem
The last comment by aegean is incorrect: if you read the first post you could notice that "the ownership seems to go to root and it's hard to tell which ones to make go away". So using ipcrm in many cases it's impossible.
I had Redhat 7.3 running on latest kernel 2.4.18-17.7.xsmp a couple weeks ago when I started getting "mm" or "libmm" shared memory errors. At least, that's what I call it since I know nothing about IPC or kernel stuff and didn't find the real source of the problem after looking all over the place the entire day. I just ended up rebooting, and selecting the previous (and currently running) kernel 2.4.18-10smp since I didn't (and still don't) know the cause of the 'mm' problem. Problems: The entire system was refusing to run lots of things such as Apache, PHP, and couldn't even log into X under root at console. For example, running 'php -v' resulted in a immediate error "failed to initialize module mm error 0" or something like that. Anyway... just having 'found' the nifty 'ipcs' command, here is my current IPC stats: # ipcs -l ------ Shared Memory Limits -------- max number of segments = 4096 max seg size (kbytes) = 32768 max total shared memory (kbytes) = 8388608 min seg size (bytes) = 1 ------ Semaphore Limits -------- max number of arrays = 128 max semaphores per array = 250 max semaphores system wide = 32000 max ops per semop call = 32 semaphore max value = 32767 ------ Messages: Limits -------- max queues system wide = 16 max size of message (bytes) = 8192 default max size of queue (bytes) = 16384 # ipcs -u ------ Shared Memory Status -------- segments allocated 6 pages allocated 16749 pages resident 63 pages swapped 0 Swap performance: 0 attempts 0 successes ------ Semaphore Status -------- used arrays = 93 allocated semaphores = 93 ------ Messages: Status -------- allocated queues = 0 used headers = 0 used space = 0 bytes ********* Does the above numbers mean my semaphore usage is at a critical level? Or is this a normal occurance? ********* # ipcs -a ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x00000000 433422336 root 600 1052672 9 dest 0x00000000 433455105 root 600 33554432 9 dest 0x00000000 433487874 root 600 33554432 9 dest 0x00000000 433520643 apache 600 46084 9 dest 0x00000000 432635908 gdm 777 196608 2 dest 0x00000000 432701445 gdm 777 196608 2 dest ------ Semaphore Arrays -------- key semid owner perms nsems status 0x00000000 7798784 apache 600 1 0x00000000 7831553 apache 600 1 [snip] 0x00000000 360459 root 600 1 0x00000000 393228 root 600 1 0x00000000 425997 root 600 1 0x00000000 458766 root 600 1 0x00000000 753687 bikram 600 1 0x00000000 786456 bikram 600 1 0x00000000 7995480 apache 600 1 0x00000000 10387545 bikram 600 1 0x00000000 10420314 bikram 600 1 0x00000000 11829339 bikram 600 1 0x00000000 11862108 bikram 600 1 [ SNIP SNIP SNIP SNIP SNIP SNIP SNIP SNIP SNIP SNIP SNIP SNIP ] [ ***** LOTS OF SEMAPHORS OWNED BY USER 'bikram' !!!! ***** ] [ SNIP SNIP SNIP SNIP SNIP SNIP SNIP SNIP SNIP SNIP SNIP SNIP ] ------ Message Queues -------- key msqid owner perms used-bytes messages (none) The user 'bikram' does LOTS of PHP development. And my system's PHP has 'session' and 'session *mm*' modules in it. User 'bikram' does a lot of testing and often CTRL+C's out of his PHP scripts. Check out the details of one 'his' semaphores: # ipcs -s -i 753687 Semaphore Array semid=753687 uid=501 gid=501 cuid=501 cgid=501 mode=0600, access_perms=0600 nsems = 1 otime = Tue Oct 29 12:25:18 2002 ctime = Tue Oct 29 12:25:10 2002 semnum value ncount zcount pid 0 0 0 0 3816 Is it normal for a semaphore to still be there a week after it's been created and the original 'pid' no longer exists? Is abnormal aborts of PHP causing all these semaphores to be left in the kernel? What happens to programs (like PHP, Apache, X) when the semaphore count has reached the limit (128 in my case) ??? If above is the cause of the problem, then what is the best route to take to solve the problem? Is in libmm's fault for not destroying the semaphore automatically? Is it PHP's fault? Is it the kernel's fault for not 'auto-cleaning' itself? Or... do I have no idea what I'm talking about? :) thanks for any help, Scott.
This should be fixed by: http://rhn.redhat.com/errata/RHBA-2002-273.html *** This bug has been marked as a duplicate of 74543 ***
Changed to 'CLOSED' state since 'RESOLVED' has been deprecated.