Bug 70846

One additional note worth pointing out that may clarify things: Both PHP and 
Apache dynamically link mm and use it for semaphore operations.

I have seen this problem as well. I have Apache alert me when it dies, and it 
seems to be dying hourly at its most frequent. This corresponds to the 
semaphore list I'm seeing:

       key      semid perms      nsems   uid   gid  cuid  cgid      otime      
ctime
         0      32769   600          1   500   500   500   500          0 
1030418965 Mon Aug 26 23:29:25 2002
         0      65538   600          1    99     0     0     0          0 
1030419002 Mon Aug 26 23:30:02 2002
         0      98307   600          1    99     0     0     0          0 
1030419002 Mon Aug 26 23:30:02 2002
         0     196612   600          1    99     0     0     0 1030421359 
1030419004 Mon Aug 26 23:30:04 2002
         0     229381   600          1    99     0     0     0 1030421359 
1030419004 Mon Aug 26 23:30:04 2002
         0     262150   600          1     0     0     0     0 1030421359 
1030419004 Mon Aug 26 23:30:04 2002
         0     294919   600          1     0     0     0     0          0 
1030419004 Mon Aug 26 23:30:04 2002
         0     327688   600          1    99    99     0     0 1030421714 
1030419005 Mon Aug 26 23:30:05 2002
         0     360457   600          1    99     0     0     0          0 
1030421361 Tue Aug 27 00:09:21 2002
         0     393226   600          1    99     0     0     0          0 
1030421361 Tue Aug 27 00:09:21 2002
         0     491531   600          1    99     0     0     0          0 
1030421714 Tue Aug 27 00:15:14 2002
         0     524300   600          1    99     0     0     0          0 
1030421714 Tue Aug 27 00:15:14 2002
         0     786445   600          1    99     0     0     0 1030449614 
1030435334 Tue Aug 27 04:02:14 2002
         0     819214   600          1    99     0     0     0 1030449614 
1030435334 Tue Aug 27 04:02:14 2002
         0     851983   600          1     0     0     0     0 1030449614 
1030435334 Tue Aug 27 04:02:14 2002
         0     884752   600          1     0     0     0     0          0 
1030435334 Tue Aug 27 04:02:14 2002
         0     917521   600          1    99    99     0     0 1030449618 
1030435334 Tue Aug 27 04:02:14 2002
         0     950290   600          1    99     0     0     0          0 
1030449619 Tue Aug 27 08:00:19 2002
         0     983059   600          1    99     0     0     0          0 
1030449619 Tue Aug 27 08:00:19 2002
         0    1015828   600          1     0     0     0     0 1030449620 
1030449620 Tue Aug 27 08:00:20 2002
         0    1048597   600          1     0     0     0     0          0 
1030449620 Tue Aug 27 08:00:20 2002
         0    1081366   600          1    99     0     0     0 1030454113 
1030449620 Tue Aug 27 08:00:20 2002
         0    1114135   600          1    99     0     0     0 1030454113 
1030449620 Tue Aug 27 08:00:20 2002
         0    1146904   600          1     0     0     0     0 1030454113 
1030449620 Tue Aug 27 08:00:20 2002
         0    1179673   600          1     0     0     0     0          0 
1030449620 Tue Aug 27 08:00:20 2002
         0    1212442   600          1    99    99     0     0 1030454118 
1030449620 Tue Aug 27 08:00:20 2002
         0    1245211   600          1    99     0     0     0          0 
1030454118 Tue Aug 27 09:15:18 2002
         0    1277980   600          1    99     0     0     0          0 
1030454118 Tue Aug 27 09:15:18 2002
         0    1310749   600          1     0     0     0     0 1030454119 
1030454119 Tue Aug 27 09:15:19 2002
         0    1343518   600          1     0     0     0     0          0 
1030454119 Tue Aug 27 09:15:19 2002
         0    1376287   600          1    99     0     0     0 1030456813 
1030454122 Tue Aug 27 09:15:22 2002
         0    1409056   600          1    99     0     0     0 1030456813 
1030454122 Tue Aug 27 09:15:22 2002
         0    1441825   600          1     0     0     0     0 1030456813 
1030454122 Tue Aug 27 09:15:22 2002
         0    1474594   600          1     0     0     0     0          0 
1030454122 Tue Aug 27 09:15:22 2002
         0    1507363   600          1    99    99     0     0 1030456818 
1030454122 Tue Aug 27 09:15:22 2002
         0    1540132   600          1    99     0     0     0          0 
1030456818 Tue Aug 27 10:00:18 2002
         0    1572901   600          1    99     0     0     0          0 
1030456818 Tue Aug 27 10:00:18 2002
         0          0   600          1    99   500   500   500          0 
1030456819 Tue Aug 27 10:00:19 2002
         0    1605670   600          1     0     0     0     0 1030456819 
1030456819 Tue Aug 27 10:00:19 2002
         0    1638439   600          1     0     0     0     0          0 
1030456819 Tue Aug 27 10:00:19 2002
         0    1671208   600          1    99     0     0     0 1030456819 
1030456819 Tue Aug 27 10:00:19 2002
         0    1703977   600          1    99     0     0     0 1030456819 
1030456819 Tue Aug 27 10:00:19 2002
         0    1736746   600          1     0     0     0     0 1030456819 
1030456819 Tue Aug 27 10:00:19 2002
         0    1769515   600          1     0     0     0     0          0 
1030456819 Tue Aug 27 10:00:19 2002
         0    1802284   600          1    99    99     0     0 1030457784 
1030456819 Tue Aug 27 10:00:19 2002

Btw, I am using all the same versions of php, apache, and libmm as above. 
However, I am running 7.3 as opposed to 7.2.

I'm seeing similar behavior under 7.1 and 7.3 with:

mm-1.1.3-8
apache-1.3.23-14
php-4.2.2-0 (that I snagged from Rawhide before it was backed out)

I'm not seeing any problems with normal operations, but PHP (and therefore
Apache itself) won't restart after running for several days. I'm also seeing
this with command-line PHP scripts I run - after a few days, they won't run
because they can't initialize the mm library (strace shows that it can't obtain
a semaphore).

Now that this has been confirmed, I'm updating it to a higher priority.

Slight correction: I'm seeing this on *7.1* with:

mm-1.1.3-8
apache-1.3.22-5.7.1
php-4.0.6-14

and *7.3* with:

mm-1.1.3-8
apache-1.3.23-14
php-4.2.2-0

This has bitten me bigtime :(

A fix to avoid having to reboot is:

for i in `ipcs -s -c | perl -pe "s/([0-9]+).+/\1/"`; do ipcrm sem $i; done

This works for me, although this could wreak havoc on the system :-S

Hope this is resolved soon, I'm in trouble over this (all our hosting went down,
we had major outage :( )

I've e-mailed nalin to get some attention for this bug, since it has 
been over three weeks without a reply from QA and people are getting bitten. 
(Nalin Dahyabhai is the most recent changelog author on the apache, mm, and php 
packages I'm using.)

[root@alpha root]# ipcs > l
[root@alpha root]# rpm -q php
php-4.1.2-7.2.4
[root@alpha root]# php -v
4.1.2
[root@alpha root]# ipcs > ll
[root@alpha root]# diff l ll
[root@alpha root]#

*** This bug has been marked as a duplicate of 71097 ***

I'm sorry, but this is not closed by any means! 

The use of kernel semaphores by mm has just made a bad problem worse. The fact
that PHP leaks semaphores just makes the problem surface faster, but it is
definitely not the cause.  

As stated above, the system only has 128 semaphore arrays total, and each apache
running uses up 4 of those. Any user can now start an apache instance, making a
DOS attack trivial. The old version of apache+mm used files instead, which were
written to /var/run, making it impossible for a non-privileged users to run an
apache process, which is a bug in itself (see
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=61030 which was totally
ignored, b.t.w.).

As we run separate httpd's for each customer on the same box, this makes hosting
 more than 32 sites on a box impossible, which is WAY below the capabilities of
the hardware. 

Please fix this properly, it is a constant source of headaches. A proposed
simple fix is included in the bugreport I referred to above. This has worked in
our hosting setup for the past year.

Reopening this bug and changing topic to refer to potential denial-of-service
issue.  Also upgrading to 'security' severity, because, as you point out, any
user and DOS the system and cause all kinds of fun problems.

I can only speak for php
mm support was *removed* because mm is no longer actively maintained and is no
longer in our current product.

in the older php sets mm support was only enabled in AS, 7.2 and 7.3 and has
since been removed in errata queued up in QA that specifically addresses this issue

php-4.1.2-2.1.5 - Advanced Server
php-4.1.2-7.0.5 - 7.0
php-4.1.2-7.1.5 - 7.1
php-4.1.2-7.2.5 - 7.2
php-4.1.2-7.3.5 - 7.3

The current rawhide version is
php-4.2.2-8.0.1 - 8.0

I have no idea when QA will get around to testing these packages. it may be some
weeks but you are welcome to ping me privately for copies

Phil
=--=

As far as I can tell, the problem with mm used to be due to mod_ssl.

There is another (better) patch for this floating around the net by
Christian Jaeger <christian.jaeger.ch> at:

http://pflanze.mine.nu/~chris/debian/apache/checkcreatemmdir.patch

This patch creates a safe mm.user directory for each user needing to run apache.
May be worth looking into.

I'm presuming this is a PHP bug until diagnosed otherwise.

Nope, a bug with PHP not releasing the semaphore array just worsens/triggers it.

The problem is using kernel semaphores in the first place, since they are a
severely limited resource that can't be tuned runtime (requires a kernel
recompile). 

At the moment one of our webservers is using 116 of the 128 semaphore arrays,
just because it is running a few instances of apache. Restarting an apache leaks
me another two arrays. In a day or two, I'll have to take down all the apache
processes, manually clean out the arrays with ipcrm and restart all the
apache's. This machine is budgetted to run at least 5 times as many apache
processes, which clearly isn't possible. 

So my options are:

1. Compile custom kernel
2. Compile custom apache+php
3. Wait for Red Hat to fix the problem

Options 1 and 2 I'd like to avoid, as the whole point of running a packaged
distribution like Red Hat is to avoid the hassle of manually compiling custom
packages.

I don't care if "null" fixes the problem. Red Hat 8 isn't even out yet, and
won't be running on our servers until 8.1 at least.  Please fix it, I'm getting
tired of living in fear.... :(

Can you file a separate bug against mm for "mm should not use semaphores by
default" then, and leave this one to cover "PHP leaks semaphores"?

Between this bug, and Bug #61030 and Bug #71097, I think the apache issue is
well spelled out.  If you need a single new bug to track the requests that a)
apache/mm use shared files rather than semaphores and b) these shared files be
placed in a location such that a non-privledged use can exec apache, I will be
happy to open one.

Compiling php with "--without-mm" seems to fix the problem (tested php 4.1.2
errata release on 7.3).  Does this have any negative implications for php, if
its not using mm?  Also tested php 4.3.0dev snapshot - it appears to be exhibit
the same problem there so I'd say its either a mm bug that php exposes or its a
bug in php with its mm routines.

Is the only thing I lose in PHP the ability to use mm sessions management in
place of files?  Since I think php.ini is defaulted to files to begin with, if
you don't need session.save_handler=mm, then this could be a potential solution
(short of upgrading to Red Hat 8.0).

The notes state that errata for PHP were in QA that don't have mm enabled (for
7.3 it would be php 4.1.2-7.3.5).  Whats the hold up on these?

As I see it, and as I think is spelled out here by others, there are really two
problems:  a bug in php that causes it to leak semaphores, which exposes a very
real DOS consequence caused by configuring mm to use semaphores.  The first bug
can be fixed by compiling php without the MM library.  The second can only be
mitigated by not using semaphores with mm.

Kevin (who's off to recompile PHP RPMs without mm and see what breaks in
production web-servers since its taking Red Hat 2 months to do this...)

It's pretty obvious there is an issue with apache and mm using semaphores 
unrelated to the php problem.  If you want a separate bug report try:
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=74543

But PLEASE can someone at RedHat get this actioned.  Even the php errata 
mentioned in this bug report has not made it out.

Hear-hear!

Is anybody giving this any attention. PHP under 7.2 is inherently unable to be
used in production for as long as this issue last!

Lars Povlsen

This is a huge problem that has bitten me on multiple production servers doing 
virtual hosting.

The silence from Red Hat on this is deafening

This has become a MAJOR disappointment for me from a company I used to hold in
high regard.

The problem first bit me at the end of August 2002. This means I've been trying
to keep our servers from breaking for the last 3 months awaiting a fix.

I've been asked to file this as a separate bug by jorton on September
26 , which I did the same day as
https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=74543, notifying him by
email of the new bugreport. This report has been ignored (status is still
"NEW"). Many others have made it clear that they have been bitten by the same
problem.

When are you going to do something about this? Is the complexity beyond you, or
are you so understaffed that you can't get around fixing high priority bugs?
Either way, you might consider hiring me, I fail to see how I could possibly do
a worse job...

An errata to fix bug 74543 is in progress and should be available soon -
apologies for the delay.

Maybe a post to BugTraq or some other high profile list might get some of
RedHat's attention.  I too am baffled by Red Hat's silence and inattention to a
serious problem.  Makes me wonder how much longer they're going to be around as
a company...  Anyone up to writing a security advisory or know someone who works
at a "security" firm that issues such advisories?  I hate to force Red Hat's
hand but three months is more than enough time to fix this bugger.  All the
arguments about open-source software bugs getting fixed faster than
closed-source software bugs are bunk on this bug...

Incidentally, bug 74543 is still state "NEW", priority "normal"...

http://people.redhat.com/jorton/7.x-mm/ for unsupported RPMs; errata currently
being prepared.

I am not sure if I should open a new bug for this or not:

Create the following simple PHP script and access it through a web browser:

<?php
  $crash = array( 0 => 2,
                 'test' => 2,
                  1 => 'hello',
                 'say' => 'hello',
                  2 => 42,
                 'life' => 42,
                  3 => 'this should help \'crash\' the machine',
                 'hoho' => 'this should help \'crash\' the machine');

  print_r($crash);

  for( $i=0; $i<count($crash); $i++ )
    $crash[$i] = stripslashes($crash[$i]);

  print_r($crash);
?>

It should die with an error similar to this:

Fatal error: Allowed memory size of 8388608 bytes exhausted (tried to allocate 
35 bytes) in /home/eid/crash.php on line 14

Reload this page a good 5-10 times.  If you run 'ipcs -s' and then restart 
apache and run 'ipcs -s' again you will find that the number of semaphore 
arrays has increased and the first few semid's are unchanged (not having been 
freed when apache shutdown?).  If you rinse and repeat this with a crude shell 
script like:

while [ true ]; do
 wget -O - http://localhost/crash.php
 wget -O - http://localhost/crash.php
 wget -O - http://localhost/crash.php
 wget -O - http://localhost/crash.php
 wget -O - http://localhost/crash.php
 wget -O - http://localhost/crash.php
 wget -O - http://localhost/crash.php
 wget -O - http://localhost/crash.php
 wget -O - http://localhost/crash.php
 wget -O - http://localhost/crash.php
 ipcs -s|grep apache|wc
 /etc/rc.d/init.d/httpd restart
 sleep 1
 ipcs -s|grep apache|wc
done

then you'll find the semaphore array numbers increasing a couple every restart 
and apache taking longer and longer to do each restart until eventually (once 
all 128 semaphore arrays are used) it refuses to start at all with the message 
reported earlier in this bug (70846):

Starting httpd: Ouch! ap_mm_create(1048576, "/var/run/httpd.mm.5619") failed
Error: MM: mm:core: failed to acquire semaphore (No space left on device): OS: 
Invalid argument
                                                           [FAILED]

Just restarting apache in a loop without loading crash.php on a freshly booted 
system does not cause the number of semaphores to spiral - it stays constant at 
5.

This is verifyable on multiple RH7.1 and a RH8.0 machine, all fully updated 
through RHN (except for kernels).

Louis

http://rhn.redhat.com/errata/RHBA-2002-273.html

has been issued to change mm back to using fcntl locks, so this leak should go
away - again, apologies it took so long for this to happen.

I'll leave this bug open as there's clearly a PHP bug here too in how it uses mm
properly.  (although it is now less serious now)

Have installed this errata http://rhn.redhat.com/errata/RHBA-2002-273.html.

Problem with apache is fixed now. When I run some PHP scripts at the 
commandline I still get:  PHP Fatal error:  Unable to start session mm module 
in Unknown on line 0

Got this problem already for a few months.

php version: 4.1.2
redhat: 7.2
mm-1.1.3-11
mm-devel-1.1.3-11

We are running:

php version: 4.1.2
redhat: 7.3
mm-1.1.3-11

and have the same problem: 

When I run some PHP scripts at the commandline as non-root user:  PHP Fatal 
error:  Unable to start session mm module 
in Unknown on line 0.  Scripts run as root user however.

This error does not occur with mm-1.1.3-8.  Please advise why this error is 
occuring with mm-1.1.3-11?

Regards,

Jonne Hannon. 

Can you attach a minimal script which reproduces that error? Also 'rpm -q php'
output.

Thanks for the report.  This is a mass bug update; since this release
of Red Hat Linux is no longer supported, please either:

a) try and reproduce the bug with a supported version of Red Hat
Enterprise Linux or Fedora Core, and re-open this bug as appropriate
after changing the Product field, or,

b) if relevant, try and reproduce this bug using the current version
of the upstream package, and report the bug upstream.

c) report the bug to the Fedora Legacy project who may wish to
continue maintenance of this package.