Bug 120515
| Summary: | No space left on device - mod_python leakes semaphores | ||
|---|---|---|---|
| Product: | [Fedora] Fedora | Reporter: | Robert Scheck <redhat-bugzilla> |
| Component: | mod_python | Assignee: | Gary Benson <gbenson> |
| Status: | CLOSED RAWHIDE | QA Contact: | |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | rawhide | CC: | astrand, axel.thimm, jorton |
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| URL: | http://search.cpan.org/~geoff/mod_perl-1.99_11/docs/user/troubleshooting/troubleshooting.pod | ||
| Whiteboard: | |||
| Fixed In Version: | 3.1.3-5 | Doc Type: | Bug Fix |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2004-10-12 14:37:57 UTC | Type: | --- |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 123268 | ||
|
Description
Robert Scheck
2004-04-09 18:17:37 UTC
Are you using mod_perl or PHP scripts which use semaphores on this machine? From doing an "service httpd restart" loop, I don't see any semaphores being leaked. Does: # ipcs -s # service httpd restart # ipcs -s show a reproducible semaphore leak on this machine? Neither mod_perl or PHP scripts are using semaphores on this machine or if it would use them, I still don't know. But the problem never occurred with httpd < 2.0.49-2 (and I had a lot of httpd restarts). It seems so, that I'm and others able to reproduce the problem, but it needs various times (most 7-15) of restarts until the problem is there: --- snipp --- # ipcs -s | perl -ane '/^0x00000000/ && `ipcrm -s $F[1]`' # # service httpd start httpd starten: [ OK ] # service httpd restart httpd beenden: [ OK ] httpd starten: [ OK ] # service httpd restart httpd beenden: [ OK ] httpd starten: [ OK ] # service httpd restart httpd beenden: [ OK ] httpd starten: [ OK ] # service httpd restart httpd beenden: [ OK ] httpd starten: [ OK ] # service httpd restart httpd beenden: [ OK ] httpd starten: [ OK ] # service httpd restart httpd beenden: [ OK ] httpd starten: [ OK ] # service httpd restart httpd beenden: [ OK ] httpd starten: [ OK ] # service httpd restart httpd beenden: [ OK ] httpd starten: [ OK ] # service httpd restart httpd beenden: [FAILED] httpd starten: [ OK ] # service httpd restart httpd beenden: [FAILED] httpd starten: [ OK ] # # ipcs -s ----- Semaphorenfelder ----- Schlüssel SemID Besitzer Rechte nsems 0x00000000 37552128 apache 600 1 0x00000000 37584897 apache 600 1 0x00000000 37617666 apache 600 1 0x00000000 37650435 apache 600 1 0x00000000 37683204 apache 600 1 0x00000000 37715973 apache 600 1 0x00000000 37748742 apache 600 1 0x00000000 37781511 apache 600 1 0x00000000 37814280 apache 600 1 0x00000000 37847049 apache 600 1 0x00000000 37879818 apache 600 1 0x00000000 37912587 apache 600 1 0x00000000 37945356 apache 600 1 0x00000000 37978125 apache 600 1 0x00000000 38010894 apache 600 1 0x00000000 38043663 apache 600 1 0x00000000 38076432 apache 600 1 0x00000000 38109201 apache 600 1 0x00000000 38141970 apache 600 1 0x00000000 38174739 apache 600 1 0x00000000 38207508 apache 600 1 0x00000000 38240277 apache 600 1 0x00000000 38273046 apache 600 1 0x00000000 38305815 apache 600 1 0x00000000 38338584 apache 600 1 0x00000000 38371353 apache 600 1 0x00000000 38404122 apache 600 1 0x00000000 38436891 apache 600 1 0x00000000 38469660 apache 600 1 0x00000000 38502429 apache 600 1 0x00000000 38535198 apache 600 1 0x00000000 38567967 apache 600 1 0x00000000 38633504 apache 600 1 0x00000000 38666273 apache 600 1 0x00000000 38699042 apache 600 1 0x00000000 38731811 apache 600 1 0x00000000 38764580 apache 600 1 0x00000000 38797349 apache 600 1 0x00000000 38830118 apache 600 1 0x00000000 38862887 apache 600 1 0x00000000 38895656 apache 600 1 0x00000000 38928425 apache 600 1 0x00000000 38961194 apache 600 1 0x00000000 38993963 apache 600 1 0x00000000 39026732 apache 600 1 0x00000000 39059501 apache 600 1 0x00000000 39092270 apache 600 1 0x00000000 39125039 apache 600 1 0x00000000 39157808 apache 600 1 0x00000000 39190577 apache 600 1 0x00000000 39223346 apache 600 1 0x00000000 39256115 apache 600 1 0x00000000 39288884 apache 600 1 0x00000000 39321653 apache 600 1 0x00000000 39354422 apache 600 1 0x00000000 39387191 apache 600 1 0x00000000 39419960 apache 600 1 0x00000000 39452729 apache 600 1 0x00000000 39485498 apache 600 1 0x00000000 39518267 apache 600 1 0x00000000 39551036 apache 600 1 0x00000000 39583805 apache 600 1 0x00000000 39616574 apache 600 1 0x00000000 39649343 apache 600 1 0x00000000 43155520 apache 600 1 0x00000000 43188289 apache 600 1 0x00000000 43221058 apache 600 1 0x00000000 43253827 apache 600 1 0x00000000 43286596 apache 600 1 0x00000000 43319365 apache 600 1 0x00000000 43352134 apache 600 1 0x00000000 43384903 apache 600 1 0x00000000 43417672 apache 600 1 0x00000000 43450441 apache 600 1 0x00000000 43483210 apache 600 1 0x00000000 43515979 apache 600 1 0x00000000 43548748 apache 600 1 0x00000000 43581517 apache 600 1 0x00000000 43614286 apache 600 1 0x00000000 43647055 apache 600 1 0x00000000 43679824 apache 600 1 0x00000000 43712593 apache 600 1 0x00000000 43745362 apache 600 1 0x00000000 43778131 apache 600 1 0x00000000 43810900 apache 600 1 0x00000000 43843669 apache 600 1 0x00000000 43876438 apache 600 1 0x00000000 43909207 apache 600 1 0x00000000 43941976 apache 600 1 0x00000000 43974745 apache 600 1 0x00000000 44007514 apache 600 1 0x00000000 44040283 apache 600 1 0x00000000 44073052 apache 600 1 0x00000000 44105821 apache 600 1 0x00000000 44138590 apache 600 1 0x00000000 44171359 apache 600 1 # --- snapp --- I can reproduce the problem with php-4.3.4-10 and -11, php-4.3.5-0, php-4.3.6RC3-0 (the last two are my own builds) and older PHP versions I don't have here. I used mod_perl-1.99_11-3, mod_perl-devel-1.99_12-1 and mod_perl-devel-1.99_13-0 (last is my own build). The onliest thing I didn't change yet, was the httpd-2.0.49-2... P.S.: Sorry for the german outputs, but I think, they are understandable ;-) Btw, do you have httpd-2.0.49-1 and that one before for me (source RPMs), that I can test it with them? Ah, it's a regression in 2.0.49 then, possibly the mod_auth_ldap changes. I'll track this down; thanks for the report, Robert. mod_python-3.1.3-1 leaks the semaphores, not httpd-2.0.49 or
something else!
--- snipp ---
# service httpd stop
httpd beenden: [ OK ]
#
# ipcs -s
----- Semaphorenfelder -----
Schlüssel SemID Besitzer Rechte nsems
#
# ls -l /etc/httpd/conf.d/*
-rw-r--r-- 1 root root 1276 Apr 18 18:39 /etc/httpd/conf.d/python.conf
#
# for i in `seq 0 15`; do
> service httpd restart
> done
httpd beenden: [FAILED]
httpd starten: [ OK ]
httpd beenden: [ OK ]
httpd starten: [ OK ]
httpd beenden: [ OK ]
httpd starten: [ OK ]
httpd beenden: [ OK ]
httpd starten: [ OK ]
httpd beenden: [ OK ]
httpd starten: [FAILED]
httpd beenden: [FAILED]
httpd starten: [FAILED]
httpd beenden: [FAILED]
httpd starten: [FAILED]
httpd beenden: [FAILED]
httpd starten: [FAILED]
httpd beenden: [FAILED]
*** CTRL+C ***
# ipcs -s
----- Semaphorenfelder -----
Schlüssel SemID Besitzer Rechte nsems
0x00000000 93618176 apache 600 1
0x00000000 93650945 apache 600 1
0x00000000 93683714 apache 600 1
0x00000000 93716483 apache 600 1
0x00000000 93749252 apache 600 1
[ And exactly 123 semaphore lines (!) more ]
#
--- snapp ---
Actual results:
rpm -e --nodeps mod_python && echo "Now everything will work fine!"
Sorry about the command above, but I'm really frustrated, because
it needed a lot of time until I located the real problem...
Okay, but I still don't know how to debug the mod_python problem, so
I would have done it already ;-)
I'm able to reproduce that problem further on having httpd 2.0.50-3, php 4.3.7-4 and mod_python-3.1.3-2 - any idea?! But now I need ca. 20-25 restarts of httpd instead of max. 15 before. Maybe you've got also a look to http://www.modpython.org/pipermail/mod_python/2003-September/014164.html BTW, I forgot to say, that the problem of leaking semaphores was introduced with mod_python 3.1.x, because mod_python 3.0.x (I tested 3.0.3 from RHEL and 3.0.4 as own build) didn't leak any semaphores using 100 (!) httpd restarts. Awaiting solution and/or fix for this annoying bug...for my personal use I downgraded to 3.0.4 until the leak in 3.1.x is fixed. My problem was also reproducable by others at #fedora-devel @ Freenode, Fedora Core 2 and Development have this semaphore leak (that's sure), the question is how to trigger out exactly and fix it - but nobody seems to interest for it?! I have a feeling the leak might be because of the init script SIGKILLing the parent after it fails to shut down quickly enough. But regardless, mod_python should not claim *32* semaphores for itself, that's pretty nuts. I'll have a look at this next week. OK, it is simply the case that if you SIGTERM httpd *whilst it is starting up*, it will shut down immediately without removing semaphores. This is hard to fix: semaphores must be explicitly removed at shutdown. Apache handles this properly once it is started up and serving requests, but in the meantime, it's not possible. So, for instance, depending on the speed of your machine, if you do: # for f in `seq 1 15`; do service httpd restart; sleep 10; done there are no semaphores leaks, since httpd has time to startup, and then when asked to shut down, it will do so cleanly. But, maybe: # for f in `seq 1 15`; do service httpd restart; sleep 1; done *will* leak semaphores, because there is not enough time for httpd to start up properly in between each startup request. Anyway, I've reduced the number of semaphores which mod_python uses by default to 4 from 32, so that should help mitigate the issues. Perhaps httpd stop/start/restart should remove the apache owned semaphores? I am seeing this too on standard FC2/x86_64 stup (w/updates) for a simple redhat/fedora public mirror (no use of any fancy mod_python stuff, but there are 35 semphores locked right from the start and restarting httpd at times of load >= 10 creates the same scnario as above, even w/o a loop). Axel, mod_python-3.1.3-5 isn't available as official FC2 update, I think. So could you try to rebuild and update to >= 3.1.3-5 and test it with that? For me it solved the problem, if it also works for you after that, Joe or Gary should push an official update for FC2. Just encountered this for the first time today. I noticed an earlier report of a similar issue at bug 83324, and at least found a nice one-liner to clear the dead sems here: http://faq.otrs.org/otrs/faq.pl?ID=4 The machine is running FC2 updated except for the latest openmotif and samba packages. I don't think I have anything exotic running under Apache. |