Description of problem: --- snipp --- # service httpd restart httpd stop: [ OK ] httpd start: [FAILED] # # ipcs -s ----- Semaphore Arrays ----- key semid owner perms nsems 0x00000000 917504 apache 600 1 0x00000000 950273 apache 600 1 0x00000000 2195458 apache 600 1 0x00000000 2228227 apache 600 1 0x00000000 2260996 apache 600 1 0x00000000 2293765 apache 600 1 0x00000000 2326534 apache 600 1 0x00000000 2359303 apache 600 1 0x00000000 2392072 apache 600 1 0x00000000 2424841 apache 600 1 0x00000000 2457610 apache 600 1 0x00000000 2490379 apache 600 1 0x00000000 2523148 apache 600 1 0x00000000 2555917 apache 600 1 0x00000000 2588686 apache 600 1 0x00000000 2621455 apache 600 1 0x00000000 2654224 apache 600 1 0x00000000 2686993 apache 600 1 0x00000000 2719762 apache 600 1 0x00000000 2752531 apache 600 1 0x00000000 2785300 apache 600 1 0x00000000 2818069 apache 600 1 0x00000000 2850838 apache 600 1 0x00000000 2883607 apache 600 1 0x00000000 2916376 apache 600 1 0x00000000 2949145 apache 600 1 0x00000000 2981914 apache 600 1 0x00000000 3014683 apache 600 1 0x00000000 3047452 apache 600 1 0x00000000 3080221 apache 600 1 0x00000000 3112990 apache 600 1 0x00000000 3145759 apache 600 1 0x00000000 3178528 apache 600 1 0x00000000 3211297 apache 600 1 0x00000000 3244066 apache 600 1 0x00000000 3276835 apache 600 1 0x00000000 12550180 apache 600 1 0x00000000 12582949 apache 600 1 0x00000000 12615718 apache 600 1 0x00000000 12648487 apache 600 1 0x00000000 12681256 apache 600 1 0x00000000 12714025 apache 600 1 0x00000000 12746794 apache 600 1 0x00000000 12779563 apache 600 1 0x00000000 12812332 apache 600 1 0x00000000 12845101 apache 600 1 0x00000000 12877870 apache 600 1 0x00000000 12910639 apache 600 1 0x00000000 12943408 apache 600 1 0x00000000 12976177 apache 600 1 0x00000000 13008946 apache 600 1 0x00000000 13041715 apache 600 1 0x00000000 13074484 apache 600 1 0x00000000 13107253 apache 600 1 0x00000000 13140022 apache 600 1 0x00000000 13172791 apache 600 1 0x00000000 13205560 apache 600 1 0x00000000 13238329 apache 600 1 0x00000000 13271098 apache 600 1 0x00000000 13303867 apache 600 1 0x00000000 13336636 apache 600 1 0x00000000 13369405 apache 600 1 0x00000000 13402174 apache 600 1 0x00000000 13434943 apache 600 1 0x00000000 13467712 apache 600 1 0x00000000 13500481 apache 600 1 0x00000000 13533250 apache 600 1 0x00000000 13566019 apache 600 1 0x00000000 13598788 apache 600 1 0x00000000 13631557 apache 600 1 0x00000000 15958086 apache 600 1 0x00000000 15990855 apache 600 1 0x00000000 16023624 apache 600 1 0x00000000 16056393 apache 600 1 0x00000000 16089162 apache 600 1 0x00000000 16121931 apache 600 1 0x00000000 16154700 apache 600 1 0x00000000 16187469 apache 600 1 0x00000000 16220238 apache 600 1 0x00000000 16253007 apache 600 1 0x00000000 16285776 apache 600 1 0x00000000 16318545 apache 600 1 0x00000000 16351314 apache 600 1 0x00000000 16384083 apache 600 1 0x00000000 16416852 apache 600 1 0x00000000 16449621 apache 600 1 0x00000000 16482390 apache 600 1 0x00000000 16515159 apache 600 1 0x00000000 16547928 apache 600 1 0x00000000 16580697 apache 600 1 0x00000000 16613466 apache 600 1 0x00000000 16646235 apache 600 1 0x00000000 16679004 apache 600 1 0x00000000 16711773 apache 600 1 0x00000000 16744542 apache 600 1 0x00000000 16777311 apache 600 1 0x00000000 16810080 apache 600 1 0x00000000 16842849 apache 600 1 0x00000000 16875618 apache 600 1 0x00000000 16908387 apache 600 1 0x00000000 16941156 apache 600 1 0x00000000 16973925 apache 600 1 0x00000000 17006694 apache 600 1 0x00000000 17039463 apache 600 1 0x00000000 19759208 apache 600 1 0x00000000 19791977 apache 600 1 0x00000000 19824746 apache 600 1 0x00000000 19857515 apache 600 1 0x00000000 19890284 apache 600 1 0x00000000 19923053 apache 600 1 0x00000000 19955822 apache 600 1 0x00000000 19988591 apache 600 1 0x00000000 20021360 apache 600 1 0x00000000 20054129 apache 600 1 0x00000000 20086898 apache 600 1 0x00000000 20119667 apache 600 1 0x00000000 20152436 apache 600 1 0x00000000 20185205 apache 600 1 0x00000000 20217974 apache 600 1 0x00000000 20250743 apache 600 1 0x00000000 20283512 apache 600 1 0x00000000 20316281 apache 600 1 0x00000000 20349050 apache 600 1 0x00000000 20381819 apache 600 1 0x00000000 20414588 apache 600 1 0x00000000 20447357 apache 600 1 0x00000000 20545662 apache 600 1 0x00000000 20578431 apache 600 1 # # ipcs -s | perl -ane '/^0x00000000/ && `ipcrm -s $F[1]`' # # service httpd start httpd start: [ OK ] # --- snapp --- In the syslog I found: --- snipp --- [crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock Configuration Failed --- snapp --- Sorry, if the error message English is not correct, because I had to retranslate the error messages and texts from German to English, so my production system is German located ;-) Version-Release number of selected component (if applicable): httpd-2.0.49-2 mod_perl-devel-1.99_12-1 php-4.3.4-11 How reproducible & Steps to Reproduce: Sorry, don't know :'-( Actual results: ipcs -s | perl -ane '/^0x00000000/ && `ipcrm -s $F[1]`' solved it, but why was that problem caused? Expected results: I don't want to have that problem again ;-))) Additional info: http://search.cpan.org/~geoff/mod_perl-1.99_11/docs/user/troubleshooting/troubleshooting.pod helped me, it describes the problem.
Are you using mod_perl or PHP scripts which use semaphores on this machine? From doing an "service httpd restart" loop, I don't see any semaphores being leaked. Does: # ipcs -s # service httpd restart # ipcs -s show a reproducible semaphore leak on this machine?
Neither mod_perl or PHP scripts are using semaphores on this machine or if it would use them, I still don't know. But the problem never occurred with httpd < 2.0.49-2 (and I had a lot of httpd restarts). It seems so, that I'm and others able to reproduce the problem, but it needs various times (most 7-15) of restarts until the problem is there: --- snipp --- # ipcs -s | perl -ane '/^0x00000000/ && `ipcrm -s $F[1]`' # # service httpd start httpd starten: [ OK ] # service httpd restart httpd beenden: [ OK ] httpd starten: [ OK ] # service httpd restart httpd beenden: [ OK ] httpd starten: [ OK ] # service httpd restart httpd beenden: [ OK ] httpd starten: [ OK ] # service httpd restart httpd beenden: [ OK ] httpd starten: [ OK ] # service httpd restart httpd beenden: [ OK ] httpd starten: [ OK ] # service httpd restart httpd beenden: [ OK ] httpd starten: [ OK ] # service httpd restart httpd beenden: [ OK ] httpd starten: [ OK ] # service httpd restart httpd beenden: [ OK ] httpd starten: [ OK ] # service httpd restart httpd beenden: [FAILED] httpd starten: [ OK ] # service httpd restart httpd beenden: [FAILED] httpd starten: [ OK ] # # ipcs -s ----- Semaphorenfelder ----- Schlüssel SemID Besitzer Rechte nsems 0x00000000 37552128 apache 600 1 0x00000000 37584897 apache 600 1 0x00000000 37617666 apache 600 1 0x00000000 37650435 apache 600 1 0x00000000 37683204 apache 600 1 0x00000000 37715973 apache 600 1 0x00000000 37748742 apache 600 1 0x00000000 37781511 apache 600 1 0x00000000 37814280 apache 600 1 0x00000000 37847049 apache 600 1 0x00000000 37879818 apache 600 1 0x00000000 37912587 apache 600 1 0x00000000 37945356 apache 600 1 0x00000000 37978125 apache 600 1 0x00000000 38010894 apache 600 1 0x00000000 38043663 apache 600 1 0x00000000 38076432 apache 600 1 0x00000000 38109201 apache 600 1 0x00000000 38141970 apache 600 1 0x00000000 38174739 apache 600 1 0x00000000 38207508 apache 600 1 0x00000000 38240277 apache 600 1 0x00000000 38273046 apache 600 1 0x00000000 38305815 apache 600 1 0x00000000 38338584 apache 600 1 0x00000000 38371353 apache 600 1 0x00000000 38404122 apache 600 1 0x00000000 38436891 apache 600 1 0x00000000 38469660 apache 600 1 0x00000000 38502429 apache 600 1 0x00000000 38535198 apache 600 1 0x00000000 38567967 apache 600 1 0x00000000 38633504 apache 600 1 0x00000000 38666273 apache 600 1 0x00000000 38699042 apache 600 1 0x00000000 38731811 apache 600 1 0x00000000 38764580 apache 600 1 0x00000000 38797349 apache 600 1 0x00000000 38830118 apache 600 1 0x00000000 38862887 apache 600 1 0x00000000 38895656 apache 600 1 0x00000000 38928425 apache 600 1 0x00000000 38961194 apache 600 1 0x00000000 38993963 apache 600 1 0x00000000 39026732 apache 600 1 0x00000000 39059501 apache 600 1 0x00000000 39092270 apache 600 1 0x00000000 39125039 apache 600 1 0x00000000 39157808 apache 600 1 0x00000000 39190577 apache 600 1 0x00000000 39223346 apache 600 1 0x00000000 39256115 apache 600 1 0x00000000 39288884 apache 600 1 0x00000000 39321653 apache 600 1 0x00000000 39354422 apache 600 1 0x00000000 39387191 apache 600 1 0x00000000 39419960 apache 600 1 0x00000000 39452729 apache 600 1 0x00000000 39485498 apache 600 1 0x00000000 39518267 apache 600 1 0x00000000 39551036 apache 600 1 0x00000000 39583805 apache 600 1 0x00000000 39616574 apache 600 1 0x00000000 39649343 apache 600 1 0x00000000 43155520 apache 600 1 0x00000000 43188289 apache 600 1 0x00000000 43221058 apache 600 1 0x00000000 43253827 apache 600 1 0x00000000 43286596 apache 600 1 0x00000000 43319365 apache 600 1 0x00000000 43352134 apache 600 1 0x00000000 43384903 apache 600 1 0x00000000 43417672 apache 600 1 0x00000000 43450441 apache 600 1 0x00000000 43483210 apache 600 1 0x00000000 43515979 apache 600 1 0x00000000 43548748 apache 600 1 0x00000000 43581517 apache 600 1 0x00000000 43614286 apache 600 1 0x00000000 43647055 apache 600 1 0x00000000 43679824 apache 600 1 0x00000000 43712593 apache 600 1 0x00000000 43745362 apache 600 1 0x00000000 43778131 apache 600 1 0x00000000 43810900 apache 600 1 0x00000000 43843669 apache 600 1 0x00000000 43876438 apache 600 1 0x00000000 43909207 apache 600 1 0x00000000 43941976 apache 600 1 0x00000000 43974745 apache 600 1 0x00000000 44007514 apache 600 1 0x00000000 44040283 apache 600 1 0x00000000 44073052 apache 600 1 0x00000000 44105821 apache 600 1 0x00000000 44138590 apache 600 1 0x00000000 44171359 apache 600 1 # --- snapp --- I can reproduce the problem with php-4.3.4-10 and -11, php-4.3.5-0, php-4.3.6RC3-0 (the last two are my own builds) and older PHP versions I don't have here. I used mod_perl-1.99_11-3, mod_perl-devel-1.99_12-1 and mod_perl-devel-1.99_13-0 (last is my own build). The onliest thing I didn't change yet, was the httpd-2.0.49-2... P.S.: Sorry for the german outputs, but I think, they are understandable ;-) Btw, do you have httpd-2.0.49-1 and that one before for me (source RPMs), that I can test it with them?
Ah, it's a regression in 2.0.49 then, possibly the mod_auth_ldap changes. I'll track this down; thanks for the report, Robert.
mod_python-3.1.3-1 leaks the semaphores, not httpd-2.0.49 or something else! --- snipp --- # service httpd stop httpd beenden: [ OK ] # # ipcs -s ----- Semaphorenfelder ----- Schlüssel SemID Besitzer Rechte nsems # # ls -l /etc/httpd/conf.d/* -rw-r--r-- 1 root root 1276 Apr 18 18:39 /etc/httpd/conf.d/python.conf # # for i in `seq 0 15`; do > service httpd restart > done httpd beenden: [FAILED] httpd starten: [ OK ] httpd beenden: [ OK ] httpd starten: [ OK ] httpd beenden: [ OK ] httpd starten: [ OK ] httpd beenden: [ OK ] httpd starten: [ OK ] httpd beenden: [ OK ] httpd starten: [FAILED] httpd beenden: [FAILED] httpd starten: [FAILED] httpd beenden: [FAILED] httpd starten: [FAILED] httpd beenden: [FAILED] httpd starten: [FAILED] httpd beenden: [FAILED] *** CTRL+C *** # ipcs -s ----- Semaphorenfelder ----- Schlüssel SemID Besitzer Rechte nsems 0x00000000 93618176 apache 600 1 0x00000000 93650945 apache 600 1 0x00000000 93683714 apache 600 1 0x00000000 93716483 apache 600 1 0x00000000 93749252 apache 600 1 [ And exactly 123 semaphore lines (!) more ] # --- snapp --- Actual results: rpm -e --nodeps mod_python && echo "Now everything will work fine!" Sorry about the command above, but I'm really frustrated, because it needed a lot of time until I located the real problem... Okay, but I still don't know how to debug the mod_python problem, so I would have done it already ;-)
I'm able to reproduce that problem further on having httpd 2.0.50-3, php 4.3.7-4 and mod_python-3.1.3-2 - any idea?! But now I need ca. 20-25 restarts of httpd instead of max. 15 before. Maybe you've got also a look to http://www.modpython.org/pipermail/mod_python/2003-September/014164.html
BTW, I forgot to say, that the problem of leaking semaphores was introduced with mod_python 3.1.x, because mod_python 3.0.x (I tested 3.0.3 from RHEL and 3.0.4 as own build) didn't leak any semaphores using 100 (!) httpd restarts. Awaiting solution and/or fix for this annoying bug...for my personal use I downgraded to 3.0.4 until the leak in 3.1.x is fixed.
My problem was also reproducable by others at #fedora-devel @ Freenode, Fedora Core 2 and Development have this semaphore leak (that's sure), the question is how to trigger out exactly and fix it - but nobody seems to interest for it?!
I have a feeling the leak might be because of the init script SIGKILLing the parent after it fails to shut down quickly enough. But regardless, mod_python should not claim *32* semaphores for itself, that's pretty nuts.
I'll have a look at this next week.
OK, it is simply the case that if you SIGTERM httpd *whilst it is starting up*, it will shut down immediately without removing semaphores. This is hard to fix: semaphores must be explicitly removed at shutdown. Apache handles this properly once it is started up and serving requests, but in the meantime, it's not possible. So, for instance, depending on the speed of your machine, if you do: # for f in `seq 1 15`; do service httpd restart; sleep 10; done there are no semaphores leaks, since httpd has time to startup, and then when asked to shut down, it will do so cleanly. But, maybe: # for f in `seq 1 15`; do service httpd restart; sleep 1; done *will* leak semaphores, because there is not enough time for httpd to start up properly in between each startup request. Anyway, I've reduced the number of semaphores which mod_python uses by default to 4 from 32, so that should help mitigate the issues.
Perhaps httpd stop/start/restart should remove the apache owned semaphores? I am seeing this too on standard FC2/x86_64 stup (w/updates) for a simple redhat/fedora public mirror (no use of any fancy mod_python stuff, but there are 35 semphores locked right from the start and restarting httpd at times of load >= 10 creates the same scnario as above, even w/o a loop).
Axel, mod_python-3.1.3-5 isn't available as official FC2 update, I think. So could you try to rebuild and update to >= 3.1.3-5 and test it with that? For me it solved the problem, if it also works for you after that, Joe or Gary should push an official update for FC2.
Just encountered this for the first time today. I noticed an earlier report of a similar issue at bug 83324, and at least found a nice one-liner to clear the dead sems here: http://faq.otrs.org/otrs/faq.pl?ID=4 The machine is running FC2 updated except for the latest openmotif and samba packages. I don't think I have anything exotic running under Apache.