This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours
Bug 120515 - No space left on device - mod_python leakes semaphores
No space left on device - mod_python leakes semaphores
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: mod_python (Show other bugs)
rawhide
All Linux
high Severity high
: ---
: ---
Assigned To: Gary Benson
http://search.cpan.org/~geoff/mod_per...
:
Depends On:
Blocks: FC3Target
  Show dependency treegraph
 
Reported: 2004-04-09 14:17 EDT by Robert Scheck
Modified: 2007-11-30 17:10 EST (History)
3 users (show)

See Also:
Fixed In Version: 3.1.3-5
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2004-10-12 10:37:57 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)

  None (edit)
Description Robert Scheck 2004-04-09 14:17:37 EDT
Description of problem:
--- snipp ---
# service httpd restart
httpd stop:                                                [  OK  ]
httpd start:                                               [FAILED]
#
# ipcs -s 
----- Semaphore Arrays -----
key        semid      owner      perms      nsems
0x00000000 917504     apache    600        1
0x00000000 950273     apache    600        1
0x00000000 2195458    apache    600        1
0x00000000 2228227    apache    600        1
0x00000000 2260996    apache    600        1
0x00000000 2293765    apache    600        1
0x00000000 2326534    apache    600        1
0x00000000 2359303    apache    600        1
0x00000000 2392072    apache    600        1
0x00000000 2424841    apache    600        1
0x00000000 2457610    apache    600        1
0x00000000 2490379    apache    600        1
0x00000000 2523148    apache    600        1
0x00000000 2555917    apache    600        1
0x00000000 2588686    apache    600        1
0x00000000 2621455    apache    600        1
0x00000000 2654224    apache    600        1
0x00000000 2686993    apache    600        1
0x00000000 2719762    apache    600        1
0x00000000 2752531    apache    600        1
0x00000000 2785300    apache    600        1
0x00000000 2818069    apache    600        1
0x00000000 2850838    apache    600        1
0x00000000 2883607    apache    600        1
0x00000000 2916376    apache    600        1
0x00000000 2949145    apache    600        1
0x00000000 2981914    apache    600        1
0x00000000 3014683    apache    600        1
0x00000000 3047452    apache    600        1
0x00000000 3080221    apache    600        1
0x00000000 3112990    apache    600        1
0x00000000 3145759    apache    600        1
0x00000000 3178528    apache    600        1
0x00000000 3211297    apache    600        1
0x00000000 3244066    apache    600        1
0x00000000 3276835    apache    600        1
0x00000000 12550180   apache    600        1
0x00000000 12582949   apache    600        1
0x00000000 12615718   apache    600        1
0x00000000 12648487   apache    600        1
0x00000000 12681256   apache    600        1
0x00000000 12714025   apache    600        1
0x00000000 12746794   apache    600        1
0x00000000 12779563   apache    600        1
0x00000000 12812332   apache    600        1
0x00000000 12845101   apache    600        1
0x00000000 12877870   apache    600        1
0x00000000 12910639   apache    600        1
0x00000000 12943408   apache    600        1
0x00000000 12976177   apache    600        1
0x00000000 13008946   apache    600        1
0x00000000 13041715   apache    600        1
0x00000000 13074484   apache    600        1
0x00000000 13107253   apache    600        1
0x00000000 13140022   apache    600        1
0x00000000 13172791   apache    600        1
0x00000000 13205560   apache    600        1
0x00000000 13238329   apache    600        1
0x00000000 13271098   apache    600        1
0x00000000 13303867   apache    600        1
0x00000000 13336636   apache    600        1
0x00000000 13369405   apache    600        1
0x00000000 13402174   apache    600        1
0x00000000 13434943   apache    600        1
0x00000000 13467712   apache    600        1
0x00000000 13500481   apache    600        1
0x00000000 13533250   apache    600        1
0x00000000 13566019   apache    600        1
0x00000000 13598788   apache    600        1
0x00000000 13631557   apache    600        1
0x00000000 15958086   apache    600        1
0x00000000 15990855   apache    600        1
0x00000000 16023624   apache    600        1
0x00000000 16056393   apache    600        1
0x00000000 16089162   apache    600        1
0x00000000 16121931   apache    600        1
0x00000000 16154700   apache    600        1
0x00000000 16187469   apache    600        1
0x00000000 16220238   apache    600        1
0x00000000 16253007   apache    600        1
0x00000000 16285776   apache    600        1
0x00000000 16318545   apache    600        1
0x00000000 16351314   apache    600        1
0x00000000 16384083   apache    600        1
0x00000000 16416852   apache    600        1
0x00000000 16449621   apache    600        1
0x00000000 16482390   apache    600        1
0x00000000 16515159   apache    600        1
0x00000000 16547928   apache    600        1
0x00000000 16580697   apache    600        1
0x00000000 16613466   apache    600        1
0x00000000 16646235   apache    600        1
0x00000000 16679004   apache    600        1
0x00000000 16711773   apache    600        1
0x00000000 16744542   apache    600        1
0x00000000 16777311   apache    600        1
0x00000000 16810080   apache    600        1
0x00000000 16842849   apache    600        1
0x00000000 16875618   apache    600        1
0x00000000 16908387   apache    600        1
0x00000000 16941156   apache    600        1
0x00000000 16973925   apache    600        1
0x00000000 17006694   apache    600        1
0x00000000 17039463   apache    600        1
0x00000000 19759208   apache    600        1
0x00000000 19791977   apache    600        1
0x00000000 19824746   apache    600        1
0x00000000 19857515   apache    600        1
0x00000000 19890284   apache    600        1
0x00000000 19923053   apache    600        1
0x00000000 19955822   apache    600        1
0x00000000 19988591   apache    600        1
0x00000000 20021360   apache    600        1
0x00000000 20054129   apache    600        1
0x00000000 20086898   apache    600        1
0x00000000 20119667   apache    600        1
0x00000000 20152436   apache    600        1
0x00000000 20185205   apache    600        1
0x00000000 20217974   apache    600        1
0x00000000 20250743   apache    600        1
0x00000000 20283512   apache    600        1
0x00000000 20316281   apache    600        1
0x00000000 20349050   apache    600        1
0x00000000 20381819   apache    600        1
0x00000000 20414588   apache    600        1
0x00000000 20447357   apache    600        1
0x00000000 20545662   apache    600        1
0x00000000 20578431   apache    600        1
#
# ipcs -s | perl -ane '/^0x00000000/ && `ipcrm -s $F[1]`'
#
# service httpd start
httpd start:                                               [  OK  ]
# 
--- snapp ---

In the syslog I found:

--- snipp ---
[crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock
Configuration Failed
--- snapp ---

Sorry, if the error message English is not correct, because I had
to retranslate the error messages and texts from German to English,
so my production system is German located ;-)

Version-Release number of selected component (if applicable):
httpd-2.0.49-2
mod_perl-devel-1.99_12-1
php-4.3.4-11

How reproducible & Steps to Reproduce:
Sorry, don't know :'-(
  
Actual results:
ipcs -s | perl -ane '/^0x00000000/ && `ipcrm -s $F[1]`'
solved it, but why was that problem caused?

Expected results:
I don't want to have that problem again ;-)))

Additional info:
http://search.cpan.org/~geoff/mod_perl-1.99_11/docs/user/troubleshooting/troubleshooting.pod 
helped me, it describes the problem.
Comment 1 Joe Orton 2004-04-13 04:40:37 EDT
Are you using mod_perl or PHP scripts which use semaphores on this
machine?  From doing an "service httpd restart" loop, I don't see any
semaphores being leaked.

Does:

# ipcs -s
# service httpd restart
# ipcs -s

show a reproducible semaphore leak on this machine?
Comment 2 Robert Scheck 2004-04-13 06:33:04 EDT
Neither mod_perl or PHP scripts are using semaphores on this machine
or if it would use them, I still don't know. But the problem never
occurred with httpd < 2.0.49-2 (and I had a lot of httpd restarts).

It seems so, that I'm and others able to reproduce the problem, but 
it needs various times (most 7-15) of restarts until the problem is 
there:

--- snipp ---
# ipcs -s | perl -ane '/^0x00000000/ && `ipcrm -s $F[1]`'
# 
# service httpd start
httpd starten:                                             [  OK  ]
# service httpd restart
httpd beenden:                                             [  OK  ]
httpd starten:                                             [  OK  ]
# service httpd restart
httpd beenden:                                             [  OK  ]
httpd starten:                                             [  OK  ]
# service httpd restart
httpd beenden:                                             [  OK  ]
httpd starten:                                             [  OK  ]
# service httpd restart
httpd beenden:                                             [  OK  ]
httpd starten:                                             [  OK  ]
# service httpd restart
httpd beenden:                                             [  OK  ]
httpd starten:                                             [  OK  ]
# service httpd restart
httpd beenden:                                             [  OK  ]
httpd starten:                                             [  OK  ]
# service httpd restart
httpd beenden:                                             [  OK  ]
httpd starten:                                             [  OK  ]
# service httpd restart
httpd beenden:                                             [  OK  ]
httpd starten:                                             [  OK  ]
# service httpd restart
httpd beenden:                                             [FAILED]
httpd starten:                                             [  OK  ]
# service httpd restart
httpd beenden:                                             [FAILED]
httpd starten:                                             [  OK  ]
#
# ipcs -s

----- Semaphorenfelder -----
Schlüssel  SemID      Besitzer   Rechte     nsems
0x00000000 37552128   apache    600        1
0x00000000 37584897   apache    600        1
0x00000000 37617666   apache    600        1
0x00000000 37650435   apache    600        1
0x00000000 37683204   apache    600        1
0x00000000 37715973   apache    600        1
0x00000000 37748742   apache    600        1
0x00000000 37781511   apache    600        1
0x00000000 37814280   apache    600        1
0x00000000 37847049   apache    600        1
0x00000000 37879818   apache    600        1
0x00000000 37912587   apache    600        1
0x00000000 37945356   apache    600        1
0x00000000 37978125   apache    600        1
0x00000000 38010894   apache    600        1
0x00000000 38043663   apache    600        1
0x00000000 38076432   apache    600        1
0x00000000 38109201   apache    600        1
0x00000000 38141970   apache    600        1
0x00000000 38174739   apache    600        1
0x00000000 38207508   apache    600        1
0x00000000 38240277   apache    600        1
0x00000000 38273046   apache    600        1
0x00000000 38305815   apache    600        1
0x00000000 38338584   apache    600        1
0x00000000 38371353   apache    600        1
0x00000000 38404122   apache    600        1
0x00000000 38436891   apache    600        1
0x00000000 38469660   apache    600        1
0x00000000 38502429   apache    600        1
0x00000000 38535198   apache    600        1
0x00000000 38567967   apache    600        1
0x00000000 38633504   apache    600        1
0x00000000 38666273   apache    600        1
0x00000000 38699042   apache    600        1
0x00000000 38731811   apache    600        1
0x00000000 38764580   apache    600        1
0x00000000 38797349   apache    600        1
0x00000000 38830118   apache    600        1
0x00000000 38862887   apache    600        1
0x00000000 38895656   apache    600        1
0x00000000 38928425   apache    600        1
0x00000000 38961194   apache    600        1
0x00000000 38993963   apache    600        1
0x00000000 39026732   apache    600        1
0x00000000 39059501   apache    600        1
0x00000000 39092270   apache    600        1
0x00000000 39125039   apache    600        1
0x00000000 39157808   apache    600        1
0x00000000 39190577   apache    600        1
0x00000000 39223346   apache    600        1
0x00000000 39256115   apache    600        1
0x00000000 39288884   apache    600        1
0x00000000 39321653   apache    600        1
0x00000000 39354422   apache    600        1
0x00000000 39387191   apache    600        1
0x00000000 39419960   apache    600        1
0x00000000 39452729   apache    600        1
0x00000000 39485498   apache    600        1
0x00000000 39518267   apache    600        1
0x00000000 39551036   apache    600        1
0x00000000 39583805   apache    600        1
0x00000000 39616574   apache    600        1
0x00000000 39649343   apache    600        1
0x00000000 43155520   apache    600        1
0x00000000 43188289   apache    600        1
0x00000000 43221058   apache    600        1
0x00000000 43253827   apache    600        1
0x00000000 43286596   apache    600        1
0x00000000 43319365   apache    600        1
0x00000000 43352134   apache    600        1
0x00000000 43384903   apache    600        1
0x00000000 43417672   apache    600        1
0x00000000 43450441   apache    600        1
0x00000000 43483210   apache    600        1
0x00000000 43515979   apache    600        1
0x00000000 43548748   apache    600        1
0x00000000 43581517   apache    600        1
0x00000000 43614286   apache    600        1
0x00000000 43647055   apache    600        1
0x00000000 43679824   apache    600        1
0x00000000 43712593   apache    600        1
0x00000000 43745362   apache    600        1
0x00000000 43778131   apache    600        1
0x00000000 43810900   apache    600        1
0x00000000 43843669   apache    600        1
0x00000000 43876438   apache    600        1
0x00000000 43909207   apache    600        1
0x00000000 43941976   apache    600        1
0x00000000 43974745   apache    600        1
0x00000000 44007514   apache    600        1
0x00000000 44040283   apache    600        1
0x00000000 44073052   apache    600        1
0x00000000 44105821   apache    600        1
0x00000000 44138590   apache    600        1
0x00000000 44171359   apache    600        1

#
--- snapp ---

I can reproduce the problem with php-4.3.4-10 and -11, php-4.3.5-0,
php-4.3.6RC3-0 (the last two are my own builds) and older PHP 
versions I don't have here. I used mod_perl-1.99_11-3,
mod_perl-devel-1.99_12-1 and mod_perl-devel-1.99_13-0 (last is my
own build).

The onliest thing I didn't change yet, was the httpd-2.0.49-2...

P.S.: Sorry for the german outputs, but I think, they are 
understandable ;-)

Btw, do you have httpd-2.0.49-1 and that one before for me (source 
RPMs), that I can test it with them?
Comment 3 Joe Orton 2004-04-13 06:40:33 EDT
Ah, it's a regression in 2.0.49 then, possibly the mod_auth_ldap
changes.  I'll track this down; thanks for the report, Robert.
Comment 4 Robert Scheck 2004-04-18 13:16:35 EDT
mod_python-3.1.3-1 leaks the semaphores, not httpd-2.0.49 or
something else!

--- snipp ---
# service httpd stop
httpd beenden:                                             [  OK  ]
#
# ipcs -s

----- Semaphorenfelder -----
Schlüssel  SemID      Besitzer   Rechte     nsems

#
# ls -l /etc/httpd/conf.d/*
-rw-r--r--    1 root     root         1276 Apr 18 18:39 /etc/httpd/conf.d/python.conf
#
# for i in `seq 0 15`; do
>   service httpd restart
> done
httpd beenden:                                             [FAILED]
httpd starten:                                             [  OK  ]
httpd beenden:                                             [  OK  ]
httpd starten:                                             [  OK  ]
httpd beenden:                                             [  OK  ]
httpd starten:                                             [  OK  ]
httpd beenden:                                             [  OK  ]
httpd starten:                                             [  OK  ]
httpd beenden:                                             [  OK  ]
httpd starten:                                             [FAILED]
httpd beenden:                                             [FAILED]
httpd starten:                                             [FAILED]
httpd beenden:                                             [FAILED]
httpd starten:                                             [FAILED]
httpd beenden:                                             [FAILED]
httpd starten:                                             [FAILED]
httpd beenden:                                             [FAILED]
*** CTRL+C ***
# ipcs -s

----- Semaphorenfelder -----
Schlüssel  SemID      Besitzer   Rechte     nsems
0x00000000 93618176   apache    600        1
0x00000000 93650945   apache    600        1
0x00000000 93683714   apache    600        1
0x00000000 93716483   apache    600        1
0x00000000 93749252   apache    600        1
[ And exactly 123 semaphore lines (!) more ]

#
--- snapp ---

Actual results:
rpm -e --nodeps mod_python && echo "Now everything will work fine!"

Sorry about the command above, but I'm really frustrated, because
it needed a lot of time until I located the real problem...

Okay, but I still don't know how to debug the mod_python problem, so
I would have done it already ;-)
Comment 5 Robert Scheck 2004-07-02 13:51:03 EDT
I'm able to reproduce that problem further on having httpd 2.0.50-3,
php 4.3.7-4 and mod_python-3.1.3-2 - any idea?!

But now I need ca. 20-25 restarts of httpd instead of max. 15 before.

Maybe you've got also a look to 
http://www.modpython.org/pipermail/mod_python/2003-September/014164.html
Comment 6 Robert Scheck 2004-07-02 16:29:49 EDT
BTW, I forgot to say, that the problem of leaking semaphores was 
introduced with mod_python 3.1.x, because mod_python 3.0.x (I tested 
3.0.3 from RHEL and 3.0.4 as own build) didn't leak any semaphores 
using 100 (!) httpd restarts.

Awaiting solution and/or fix for this annoying bug...for my personal 
use I downgraded to 3.0.4 until the leak in 3.1.x is fixed.
Comment 7 Robert Scheck 2004-09-06 14:14:09 EDT
My problem was also reproducable by others at #fedora-devel @ Freenode,
Fedora Core 2 and Development have this semaphore leak (that's sure), 
the question is how to trigger out exactly and fix it - but nobody 
seems to interest for it?!
Comment 8 Joe Orton 2004-09-17 18:29:57 EDT
I have a feeling the leak might be because of the init script
SIGKILLing the parent after it fails to shut down quickly enough.  But
regardless, mod_python should not claim *32* semaphores for itself,
that's pretty nuts.
Comment 9 Joe Orton 2004-09-17 18:30:31 EDT
I'll have a look at this next week.
Comment 10 Joe Orton 2004-10-12 10:37:57 EDT
OK, it is simply the case that if you SIGTERM httpd *whilst it is
starting up*, it will shut down immediately without removing
semaphores.  This is hard to fix: semaphores must be explicitly
removed at shutdown.  Apache handles this properly once it is started
up and serving requests, but in the meantime, it's not possible.

So, for instance, depending on the speed of your machine, if you do:

# for f in `seq 1 15`; do service httpd restart; sleep 10; done

there are no semaphores leaks, since httpd has time to startup, and
then when asked to shut down, it will do so cleanly.

But, maybe:

# for f in `seq 1 15`; do service httpd restart; sleep 1; done

*will* leak semaphores, because there is not enough time for httpd to
start up properly in between each startup request.

Anyway, I've reduced the number of semaphores which mod_python uses by
default to 4 from 32, so that should help mitigate the issues.
Comment 11 Axel Thimm 2004-11-08 15:16:07 EST
Perhaps httpd stop/start/restart should remove the apache owned
semaphores?

I am seeing this too on standard FC2/x86_64 stup (w/updates) for a
simple redhat/fedora public mirror (no use of any fancy mod_python
stuff, but there are 35 semphores locked right from the start and
restarting httpd at times of load >= 10 creates the same scnario as
above, even w/o a loop).
Comment 12 Robert Scheck 2004-11-08 15:35:45 EST
Axel, mod_python-3.1.3-5 isn't available as official FC2 update, I 
think. So could you try to rebuild and update to >= 3.1.3-5 and test 
it with that? For me it solved the problem, if it also works for you 
after that, Joe or Gary should push an official update for FC2.
Comment 13 Kenneth Porter 2004-12-01 21:00:10 EST
Just encountered this for the first time today. I noticed an earlier
report of a similar issue at bug 83324, and at least found a nice
one-liner to clear the dead sems here:

http://faq.otrs.org/otrs/faq.pl?ID=4

The machine is running FC2 updated except for the latest openmotif and
samba packages. I don't think I have anything exotic running under Apache.

Note You need to log in before you can comment on or make changes to this bug.