Bug 72158 - Apache start error
Apache start error
Status: CLOSED DUPLICATE of bug 74543
Product: Red Hat Linux
Classification: Retired
Component: apache (Show other bugs)
7.3
i386 Linux
medium Severity high
: ---
: ---
Assigned To: Nalin Dahyabhai
Brian Brock
not available...
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2002-08-21 12:46 EDT by Bill Cronin
Modified: 2007-04-18 12:45 EDT (History)
4 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2006-02-21 13:49:26 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Bill Cronin 2002-08-21 12:46:22 EDT
From Bugzilla Helper:
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Q312461; .NET 
CLR 1.0.3705)

Description of problem:
Apache will not start...

after typing service httpd start (or restart) I get...

Starting httpd: Ouch! ap_mm_create(1048576, "/var/run/httpd.mm.29690") failed
Error: MM: mm:core: failed to acquire semaphore (No space left on device): OS: 
Identifier removed

df -h reveals adequete space on all devices.  I think the problem may be that 
apache is not properly releasing semaphores when stopping.

ipcs -u...
[root@sandbox2 root]# ipcs -u

------ Shared Memory Status --------
segments allocated 4
pages allocated 16653
pages resident  15
pages swapped   0
Swap performance: 0 attempts     0 successes

------ Semaphore Status --------
used arrays = 48
allocated semaphores = 48

------ Messages: Status --------
allocated queues = 0
used headers = 0
used space = 0 bytes

service httpd restart [OK][OK] (paraphrased)...

[root@sandbox2 root]# ipcs -u

------ Shared Memory Status --------
segments allocated 4
pages allocated 16653
pages resident  15
pages swapped   0
Swap performance: 0 attempts     0 successes

------ Semaphore Status --------
used arrays = 50
allocated semaphores = 50

------ Messages: Status --------
allocated queues = 0
used headers = 0
used space = 0 bytes

Notice Used Arrays and Allocated Semaphores.  They will go up by 2 until there 
ain't no more.  ipcs -s shows only 2 owned by apache at any given time.  None 
if apache is stopped.  I'll attach the exact results later (ipcs -s) if need be.

service httpd graceful

and 

service httpd reload

don't seem to have the same problem.  Graceful and mod_perl don't seem to play 
nice, so that's not really an option.  I can ipcrm some semaphores, but as the 
ownership seems to go to root it's hard to tell which ones to make go away.  



Version-Release number of selected component (if applicable):
apache-1.3.23-14
kernel-smp-2.4.18-5
mm-1.1.3-8
mod_perl-1.26-5
mod_ssl-2.8.7-6
php-4.1.2-7.3.3
php-mysql-4.1.2-7.3.3
php-devel-4.1.2-7.3.3
php-ldap-4.1.2-7.3.3
php-imap-4.1.2-7.3.3

Let me know if any other version information is necessary.


How reproducible:
Didn't try

Steps to Reproduce:
1.I could restart apache over and over until the semaphores run out - that's 
bad.

2. ipcs -u semaphores used = N
3. service httpd restart
4. ipcs -u semaphores used = N+2

Actual Results:  1. ipcs -u semaphores used = N
2. service httpd restart
3. ipcs -u semaphores used = N+2

Expected Results:  1. ipcs -u semaphores used = N
2. service httpd restart
3. ipcs -u semaphores used = N

Additional info:

Seems to be some chat about this on the net, but nobody's getting any answers.  
Maybe mm upgrade related - hadn't noticed before then?

apache-1.3.23-14
kernel-smp-2.4.18-5
mm-1.1.3-8
mod_perl-1.26-5
mod_ssl-2.8.7-6
php-4.1.2-7.3.3
php-mysql-4.1.2-7.3.3
php-devel-4.1.2-7.3.3
php-ldap-4.1.2-7.3.3
php-imap-4.1.2-7.3.3

Let me know if any other version information is necessary.
Comment 1 Bill Cronin 2002-08-21 13:00:25 EDT
Forgot to mention a very important detail.  Rebooting fixes the problem for 
several days, and there is a custom log rotation script that runs nightly and 
condrestarts apache.  I'll check it out next time things are broken to get the 
semaphore count.
Comment 2 Darren Gamble 2002-08-26 13:40:05 EDT
I have exactly the same problem, on Red Hat 7.2.  We recently did an up2date 
which included the new mm package; this is probably related.

Rebooting the machine did fix the problem, although it hasn't been running long 
enough to see if the problem shows up again due to log rotations.
Comment 3 ctolley 2002-08-28 16:47:53 EDT
I just ran into this problem today during a server migration.  It's a shared 
virutal environment, so I was setting up separate instances of httpd to run as 
the shared UID/GID of the site instead of apache/apache and remove the need to 
do suexec calls (and be able to run mod_php safely).  Everything was working 
fine as I added and stared up sites until this happened, then everything died.  
I hadn't found this bug report yet, so I rebooted and all was well again.

Since I'm running my httpd processes as the individual users, I may have more 
luck using identifying semaphores and using ipcrm to clean them, but I'm 
looking forward to a fix on this.  Perhaps a temporary fix would be recompiling 
apache to not use mm?  What would be the impact of recompiling without 
EAPI_MM_CORE_PATH and EAPI_MM defined?  Would this cause more problems or poor 
performance?

While this doesn't fix the problem, you can change your logrotation script so 
that it doesn't need to HUP/USR1 Apache during logrotation by 
using "copytruncate" in your logrotation script.  Here's a copy of 
my /etc/logrotate.d/apache script:

###

/var/log/httpd/*log {
    daily
    rotate 4
    copytruncate
    compress
}

###
Comment 4 Joshua M. Thompson 2002-08-28 23:51:44 EDT
Just caught this bug today too in a server I'm readying for productin use, an 
Athlon MP 1800 machine (single CPU on a Tyan Thunder K7 board) running RH 7.3. 
I've been doing a lot of web server restarts because i'm developing some custom 
mod_perl modules and I finally had to reboot beacause apache just wouldn't 
start.

This needs fixed ASAP!
Comment 5 ctolley 2002-08-29 02:21:25 EDT
I have apache packages (1.3.26) and php packages (4.1.2) compiled without mm 
support.  I'll let everyone know how it performs.  I'm still playing with RH 
7.2, so my compiles that platform, but they should install on 7.3 just fine.  
If anyone wants my SRPMS or RPMS, just drop me an email.
Comment 6 ctolley 2002-08-29 04:58:51 EDT
Forgot to mention a recompiled version of mod_ssl minus MM (2.8.9).
Comment 7 Cheng S. Wang 2002-08-30 11:00:38 EDT
We have the same problem after we've applied the mm-1.1.3-8 fix.  This happens
to several machines where httpd will be restarted often.  The problem will
disappear after reboot.
Comment 8 ctolley 2002-08-31 07:35:52 EDT
I ended up dumping my multi virt/httpd config and just started using safe_mod 
and open_basedir directives.  At any rate, my non-MM compiled versions of 
Apache+mod_ssl+php use 1 semaphore (for the parent process), but none of the 
children use any semaphores.

I'm going to run some tests on another server, restarting httpd many times to 
see if it leaks semaphores.  I'll let you know the results later next week 
unless a fix is found before then.
Comment 9 Jonathon Padfield 2002-09-16 00:20:51 EDT
Just ran across the same problem after installing the mm-1.1.3-8 update on a
Redhat 7.3 system.

The used arrays and allocated semaphores go up by two every time apache is
restarted. As this occurs on a development box, apache can be restarted fairly
frequently.

Any further ideas on the problem or a fix yet?
Comment 10 Bill Cronin 2002-10-07 08:32:19 EDT
mm-1.1.3-4 will "solve" the problem.  MM from source is currently at 1.2.1 - 
http://www.ossp.org/pkg/lib/mm/.

-Bill
Comment 11 aegean 2002-11-06 12:50:48 EST
You can use your apache server after use this command line.

ipcs -a|grep apache|awk -F\  '{ print  }' |xargs ipcrm sem
Comment 12 Need Real Name 2002-11-07 08:28:45 EST
The last comment by aegean is incorrect: if you read the first post you could 
notice that "the ownership seems to go to root and it's hard to tell which ones 
to make go away".

So using ipcrm in many cases it's impossible.
Comment 13 Scott Copus 2002-11-09 04:38:05 EST
I had Redhat 7.3 running on latest kernel 2.4.18-17.7.xsmp a couple weeks ago
when I started getting "mm" or "libmm" shared memory errors.  At least, that's
what I call it since I know nothing about IPC or kernel stuff and didn't find
the real source of the problem after looking all over the place the entire day.
 I just ended up rebooting, and selecting the previous (and currently running)
kernel 2.4.18-10smp since I didn't (and still don't) know the cause of the 'mm'
problem.

Problems:  The entire system was refusing to run lots of things such as Apache,
PHP, and couldn't even log into X under root at console.  For example, running
'php -v' resulted in a immediate error "failed to initialize module mm error 0"
or something like that.

Anyway... just having 'found' the nifty 'ipcs' command, here is my current IPC
stats:

# ipcs -l

------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 32768
max total shared memory (kbytes) = 8388608
min seg size (bytes) = 1

------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767

------ Messages: Limits --------
max queues system wide = 16
max size of message (bytes) = 8192
default max size of queue (bytes) = 16384

# ipcs -u

------ Shared Memory Status --------
segments allocated 6
pages allocated 16749
pages resident  63
pages swapped   0
Swap performance: 0 attempts     0 successes

------ Semaphore Status --------
used arrays = 93
allocated semaphores = 93

------ Messages: Status --------
allocated queues = 0
used headers = 0
used space = 0 bytes

*********
Does the above numbers mean my semaphore usage is at a critical level?  Or is
this a normal occurance?
*********

# ipcs -a

------ Shared Memory Segments --------
key        shmid      owner      perms      bytes      nattch     status
0x00000000 433422336  root      600        1052672    9          dest
0x00000000 433455105  root      600        33554432   9          dest
0x00000000 433487874  root      600        33554432   9          dest
0x00000000 433520643  apache    600        46084      9          dest
0x00000000 432635908  gdm       777        196608     2          dest
0x00000000 432701445  gdm       777        196608     2          dest

------ Semaphore Arrays --------
key        semid      owner      perms      nsems      status
0x00000000 7798784    apache    600        1
0x00000000 7831553    apache    600        1
[snip]
0x00000000 360459     root      600        1
0x00000000 393228     root      600        1
0x00000000 425997     root      600        1
0x00000000 458766     root      600        1
0x00000000 753687     bikram    600        1
0x00000000 786456     bikram    600        1
0x00000000 7995480    apache    600        1
0x00000000 10387545   bikram    600        1
0x00000000 10420314   bikram    600        1
0x00000000 11829339   bikram    600        1
0x00000000 11862108   bikram    600        1
[ SNIP SNIP SNIP SNIP SNIP SNIP SNIP SNIP SNIP SNIP SNIP SNIP ]
[ *****  LOTS OF SEMAPHORS OWNED BY USER 'bikram' !!!!  ***** ]
[ SNIP SNIP SNIP SNIP SNIP SNIP SNIP SNIP SNIP SNIP SNIP SNIP ]

------ Message Queues --------
key        msqid      owner      perms      used-bytes   messages
(none)

The user 'bikram' does LOTS of PHP development.  And my system's PHP has
'session' and 'session *mm*' modules in it.  User 'bikram' does a lot of testing
and often CTRL+C's out of his PHP scripts.  Check out the details of one 'his'
semaphores:

# ipcs -s -i 753687

Semaphore Array semid=753687
uid=501  gid=501         cuid=501        cgid=501
mode=0600, access_perms=0600
nsems = 1
otime = Tue Oct 29 12:25:18 2002
ctime = Tue Oct 29 12:25:10 2002
semnum     value      ncount     zcount     pid
0          0          0          0          3816

Is it normal for a semaphore to still be there a week after it's been created
and the original 'pid' no longer exists?  Is abnormal aborts of PHP causing all
these semaphores to be left in the kernel?  What happens to programs (like PHP,
Apache, X) when the semaphore count has reached the limit (128 in my case) ???

If above is the cause of the problem, then what is the best route to take to
solve the problem?  Is in libmm's fault for not destroying the semaphore
automatically?  Is it PHP's fault?  Is it the kernel's fault for not
'auto-cleaning' itself?

Or... do I have no idea what I'm talking about?  :)

thanks for any help,
Scott.
Comment 14 Joe Orton 2002-12-13 11:49:04 EST
This should be fixed by: http://rhn.redhat.com/errata/RHBA-2002-273.html



*** This bug has been marked as a duplicate of 74543 ***
Comment 15 Red Hat Bugzilla 2006-02-21 13:49:26 EST
Changed to 'CLOSED' state since 'RESOLVED' has been deprecated.

Note You need to log in before you can comment on or make changes to this bug.