Bug 235517 - php stopping half way, closer look reveals null terminators
php stopping half way, closer look reveals null terminators
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Linux 4
Classification: Red Hat
Component: kernel (Show other bugs)
4.4
All Linux
medium Severity medium
: ---
: ---
Assigned To: Kernel Maintainer List
Martin Jenner
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2007-04-06 12:21 EDT by James Jhurani
Modified: 2010-10-22 10:13 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2010-06-07 00:59:55 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
sysreport from server exhibiting this bug (302.12 KB, application/x-bzip)
2007-04-10 09:29 EDT, James Jhurani
no flags Details
strace string size 2048 (970.00 KB, application/x-tar)
2007-04-20 13:18 EDT, James Jhurani
no flags Details
strace string size 8192 (1.13 MB, application/x-tar)
2007-04-20 13:19 EDT, James Jhurani
no flags Details

  None (edit)
Description James Jhurani 2007-04-06 12:21:34 EDT
Description of problem:
php stops loading half way through a page. (don't worry it gets much more detailed)

Version-Release number of selected component (if applicable):
All versions of php(primarily 5.2.1), on apache 2.0.52, with RHEL 4.4 

How reproducible:
to TEST, you can simply wget http://209.85.88.179/phpinfo.php. Loading it in a
browser will just stop half way... 
  
Actual results:
you will have to stop the wget, or it will keep going. When you cat the file,
use cat -A, you will see quite a few null terminators(eg: "^@") appended to the
end of the file. 


Expected results:
phpinfo() output...

Additional info:
We have had 4-5 completely unrelated customers with this issue in the past week
or two. I roughly documented it a few days ago.

NOTE: I have changed the customers names, and domain names. 

When visiting a site, the page will load half way, and then stop. If you
download the php file with wget, you will get a big file.

webtech@10-33-3-144 ~ > wget http://moody[snip].com/test.php
--14:54:36-- http://moody[snip].com/test.php
           => `test.php.4'
Resolving moody[snip].com... 67.15.xxx.34
Connecting to moody[snip].com|67.15.xxx.34|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
    [ <=> ] 139,210 --.--K/s
14:54:36 (2.34 MB/s) - `test.php.4' saved [139210]

webtech@10-33-3-144 ~ > wget http://moody[snip].com/test.php
--14:55:27-- http://moody[snip].com/test.php
           => `test.php.5'
Resolving moody[snip].com... 67.15.xxx.34
Connecting to moody[snip].com|67.15.xxx.34|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
    [ <=> ] 31,166,083 2.24M/s
webtech@10-33-3-144 ~ >

As you can see, downloading it once, it was ~139KB. and the second time it was
~31MB. Why the different size?

note, to actually see the null characters, you need to use "cat -A file", or
"pico -w file".

 This program is free software; you can redistribute it and/or modify it under
the terms of the PHP License as published by the PHP Group and included in the
distribution in the file: LICENSE$
$
This program is distributed in the hope that it will be useful, but WITHOUT ANY
WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.$
$
If you did not receive a copy of the PHP license, or have any questions about
PHP licensing, please contact license@php.net.$
$
$
$
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

Notice the null terminators that fill the end of the file... These are what
attribute for the variable length.

The only thing we have found so far that coincides with the nulls, are bad tcp
checksums:
13:44:46.555361 IP (tos 0x0, ttl 64, id 3309, offset 0, flags [DF], proto 6,
length: 1500) 67.15.xxx.34.http > 216.12.xxx.9.44207: . 37649:39097(1448) ack
116 win 1448 <nop,nop,timestamp 99939270 25205509>
13:44:46.555596 IP (tos 0x0, ttl 64, id 3311, offset 0, flags [DF], proto 6,
length: 1500) 67.15.xxx.34.http > 216.12.xxx.9.44207: . 39097:40545(1448) ack
116 win 1448 <nop,nop,timestamp 99939270 25205510>
13:44:46.555600 IP (tos 0x0, ttl 64, id 3313, offset 0, flags [DF], proto 6,
length: 1500) 67.15.xxx.34.http > 216.12.xxx.9.44207: . 40545:41993(1448) ack
116 win 1448 <nop,nop,timestamp 99939270 25205510>
13:44:46.555971 IP (tos 0x0, ttl 64, id 3315, offset 0, flags [DF], proto 6,
length: 1500) 67.15.xxx.34.http > 216.12.xxx.9.44207: . 41993:43441(1448) ack
116 win 1448 <nop,nop,timestamp 99939270 25205510>
13:44:46.555975 IP (tos 0x0, ttl 64, id 3317, offset 0, flags [DF], proto 6,
length: 1500) 67.15.xxx.34.http > 216.12.xxx.9.44207: P 43441:44889(1448) ack
116 win 1448 <nop,nop,timestamp 99939270 25205510>
13:44:46.556480 IP bad-len 0
13:44:46.556723 IP bad-len 0
13:44:46.560095 IP bad-len 0
13:44:46.560100 IP bad-len 0
13:44:46.560103 IP bad-len 0
13:44:46.560106 IP bad-len 0
13:44:46.560109 IP bad-len 0
13:44:46.560111 IP bad-len 0
13:44:46.560114 IP bad-len 0
13:44:46.560116 IP bad-len 0
13:44:46.576212 IP bad-len 0

This one was very out right.

Others will look more like
4:47:51.191474 IP (tos 0x0, ttl 59, id 62788, offset 0, flags [DF], proto 6,
length: 52) theplanet.com.8661 > pr3.action[snip].com.http: . [tcp sum ok]
113:113(0) ack 68057 win 7740 <nop,nop,timestamp 27937994 641323617>
14:47:51.191484 IP (tos 0x0, ttl 64, id 28144, offset 0, flags [DF], proto 6,
length: 1500, bad cksum 0 (->40c)!) pr3.action[snip].com.http >
theplanet.com.8661: . 68057:69505(1448) ack 113 win 1448 <nop,nop,timestamp
641323626 27937994>
14:47:51.191488 IP (tos 0x0, ttl 64, id 28147, offset 0, flags [DF], proto 6,
length: 1500, bad cksum 0 (->409)!) pr3.action[snip].com.http >
theplanet.com.8661: P 70953:72401(1448) ack 113 win 1448 <nop,nop,timestamp
641323626 27937994>
14:47:51.191491 IP (tos 0x0, ttl 64, id 28150, offset 0, flags [DF], proto 6,
length: 1500, bad cksum 0 (->406)!) pr3.action[snip].com.http >
theplanet.com.8661: . 73849:75297(1448) ack 113

This one had bad checksums, but not length 0. These bad checksums directly
coincide with the nulls.

some other symptoms, if you wget it from itself, the file will transfer fine.

[root@moody[snip] ~]# wget http://moody[snip].com/test.php
--15:07:37-- http://moody[snip].com/test.php
           => `test.php.1'
Resolving moody[snip].com... 67.15.xxx.34
Connecting to moody[snip].com|67.15.xxx.34|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
    [ <=> ] 37,633 --.--K/s
15:07:37 (33.79 MB/s) - `test.php.1' saved [37633]
[root@moody[snip] ~]#

This happens most of the time, however every now and then you will get the nulls.

In james Ds ticket, we tested it from one server in his rack to another, and it
contained the nulls.

In all the tickets we have seen thus far, this issue affected ONLY .php files.
(italian.* is just an english -> italian dictionary file that popped up in google)
on the server:
[root@moody[snip] moodys]# md5sum italian.txt
39637589eaed62612a97f721c8b42cab italian.txt
[root@moody[snip] moodys]# md5sum italian.php
39637589eaed62612a97f721c8b42cab italian.php
[root@moody[snip] moodys]# md5sum italian.html
39637589eaed62612a97f721c8b42cab italian.html
[root@moody[snip] moodys]#

after wgetting them to my workstation from the web server:
webtech@10-33-3-144 ~ > md5sum italian.txt
39637589eaed62612a97f721c8b42cab italian.txt
webtech@10-33-3-144 ~ > md5sum italian.html
39637589eaed62612a97f721c8b42cab italian.html
webtech@10-33-3-144 ~ >

and as we expected:
webtech@10-33-3-144 ~ > md5sum italian.php
657f7ebbb4269fa16b82e6cd80a19769 italian.php
webtech@10-33-3-144 ~ >

all but .php are the same.

This rules out the switch, since it could not differentiate between file extentions.

Now each one seems to have its own little quarks. Such as james Ds would work if
you wget from the machine locally. Whereas James Hs will not. Although Sherri
H's actually affects ALL files. .txt, .html, .php, etc. I even tried .conf.

The reason I feel it is related is because they all work if you change the port
apache runs on. Yes, I know, it makes NO sense. But if you put apache on port
81, 10, 8000, as long as its NOT port 80. php will transfer the file without a
problem.

server:
[root@moody[snip] moodys]# md5sum italian.php
39637589eaed62612a97f721c8b42cab italian.php
[root@moody[snip] moodys]#
 

workstation:
webtech@10-33-3-144 ~ > wget http://moody[snip].com:81/italian.php
--15:44:45-- http://moody[snip].com:81/italian.php
           => `italian.php'
Resolving moody[snip].com... 67.15.xxx.34
Connecting to moody[snip].com|67.15.xxx.34|:81... failed: Connection refused.
webtech@10-33-3-144 ~ > wget http://moody[snip].com:81/italian.php
--15:44:56-- http://moody[snip].com:81/italian.php
           => `italian.php'
Resolving moody[snip].com... 67.15.xxx.34
Connecting to moody[snip].com|67.15.xxx.34|:81... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
    [ <=> ] 128,736 --.--K/s
15:44:56 (2.40 MB/s) - `italian.php' saved [128736]
webtech@10-33-3-144 ~ > md5sum italian.php
39637589eaed62612a97f721c8b42cab italian.php
webtech@10-33-3-144 ~ >

The md5s now match because it was downloaded from port 81.

Now with this inconsistency it was brought up that, maybe its a power issue...
to put that to rest:

I used:
for i in `cat italian.php` ; do wget .... -O /dev/null ; done

I always had to stop it for port 80, or it would just keep downloading. I let it
go 5 times, as you can see it was different every time.
webtech@10-33-3-144 ~ > cat port80 | grep K/s | sort | uniq -c
    [ <=> ] 21,333,586 6.38M/s
    [ <=> ] 7,666,987 4.10M/s
    [ <=> ] 9,350,313 6.35M/s
    [ <=> ] 7,803,474 6.20M/s
    [ <=> ] 11,582,754 6.14M/s

Now the same thing except downloaded from port 81, i let it run for about 15
seconds, and it downloaded 61 times.
webtech@10-33-3-144 ~ > cat port81 | grep K/s | wc -l
61
webtech@10-33-3-144 ~ >

Out of those 61 times, it did not fail once.
webtech@10-33-3-144 ~ > cat port81 | grep K/s | sort | uniq -c
    [ <=> ] 128,736 --.--K/s
webtech@10-33-3-144 ~ >

This would put to rest any possible i/o errors from the drive, or power issues
causing the problem.

Things we tried/ruled out:
NIC - Putting an intel NIC in the server, and disabling the on board NIC, it
still occurred.
hard drive - works on port 81 - occrring on multiple servers, no io errors
switch - only affects .php files - shouldnt be able to differentiate ports -
happens in racks, and server that are on the floor.
apache - works on port 81 - was reinstalled several times
power issues - works on port 81 - no io errors(even after hammering it)
php - works on port 81 - was reinstalled several times, and different versions

In one instance on server was swapped to opteron hardware, but it was also dual
core. One person said they switched to a single core server, and it worked fine.

However I previously tried running a uniprocessor kernel on the dual core
servers did not fix it.

Update:
Yesterday, another tech, Ronnie K. noticed that if you get with HTTP/1.0 it will
send the entire page, but HTTP/1.1 and it will not. If you do this manually via
telnet, it will not have ANY bad checksums. But if you use curl -0(to signify
HTTP/1.0) it will have the bad checksums, but the page will still transfer in
its entirety. 

Update:
It has been speculated that this is due to a rootkit that installs a kernel
module. However that is proved incorrect since we have tried using/reinstalling
other kernels, which would use different modules. 

Any insight you guys have on this issue is helpful. If you need a systat, or any
other information, just let me know.
Comment 1 James Jhurani 2007-04-06 12:33:53 EDT
Ronnie K's Update:

I've installed lighttpd-1.4.13 on 209.85.xx.180, one of James Ds boxes. It's
having the same issues as Apache. I can't view http://209.85.xx.180/phpinfo.php
on port 80, but 81 or anything else is no problem. On this server, you can view
the .php pages with HTTP 1.0, you just get badchecksum errors on tcpdump.
However, if you do this:

rwkendrick@10-33-3-139 ~ $ telnet 209.85.xx.180 80
Trying 209.85.xx.180...
Connected to 209.85.xx.180.
Escape character is '^]'.
GET /phpinfo.php HTTP/1.0

The whole file comes through with no chksum errors. Not sure why that is.
However, James is working on another server (209.85.(xx+33).99/gabetest.php) and
HTTP 1.0 doesn't work at all. So, that throws out the HTTP 1.1 theory. These
files work without any problems locally. Not using the loopback address, but the
actual IP. However, I doubt it's the network because we have a test box the
customer hasn't touched and it servers phpinfo no problem.

I was going to rebuild the kernel modules, but James already installed and
tested new kernels without any success. Read the rest of the internal posts if
you want to see all the methods we've exhausted. If you have any other ideas,
let us know.


NOTE: his references to James, are referring to me(jjhurani), not the customers
James D, or James H. unless he added the last initial. 

Comment 2 James Jhurani 2007-04-10 09:29:02 EDT
Created attachment 152131 [details]
sysreport from server exhibiting this bug
Comment 3 James Jhurani 2007-04-10 09:44:00 EDT
Note regarding the sysreport, I used the sysreport from a server that is
exhibiting the problem, but with the least amount of trouble shooting done. I
did this with the intentions of providing you with a clean sysreport with the
least amount of configuration changes made by us. 

The php files we have been testing all contain:
[root@rpweb1 public_html]# cat test.php
<?php

phpinfo();

?>

[root@rpweb1 public_html]#

Some php files with small output will work. With test.php, while the code is
small, the phpinfo output is fairly large. The output here seems to be what
counts(to cause the problem). When using a php file that simply counts line by
line, the result would vary. In some cases it would reach line 129, in other
cases 157, basically the size of what is successfully output is seemingly random. 

Please note, we have tried default configuration, and several different versions
of apache/php. I seriously doubt the sysreport will give you enough to reproduce
this issue. 

The only thing we have noticed in common between the servers, is that they are
all RHEL4.4. 


Comment 4 Joe Orton 2007-04-11 05:19:43 EDT
Thanks for the report.  So in summary the symptoms of failure you have are:

1) IP checksum/corruption failures shown in tcpdump
2) HTTP response corruption
3) occurs only on traffic from port 80
4) occurs with both httpd and lighttpd
5) occurs on multiple machines

("curl" is better at detecting HTTP chunked response corruption that "wget" FWIW)

I would suggest the following possible causes:

a) TCP filtering gone wrong at kernel level (is iptables in use?)
b) kernel bug, or more likely, as you suggested: kernel compromised
c) TCP filtering done on port 80 somewhere external to the box

You should be able to eliminate (a) and (b) by doing fresh-from-CD RHEL installs
on the boxes.  You can eliminate (c) by plugging something directly into the NIC
on the box and testing directly without passing packets through the LAN.

I'd strongly recommend you contact Red Hat Support to help with a successful
diagnosis of a complex issue like this.
Comment 5 James Jhurani 2007-04-14 04:02:55 EDT
I apologize for the delay in my response. 

I originally used curl, but simply received the error you mentioned. I believe
it was "chunky parse error". Another tech mentioned wget would keep downloading.
After some investigation with wget, we came to the aforementioned results.

In response to your suggestions:

a) iptables was stopped during the investigation.

b)  
I do not believe this is the issue. This type of rootkit generally installs an
LKM, or modifies the address of an existing one. Since we installed a new
kernel, this would have a different set of LKMs(I believe). The rootkit might be
smart enough to install into all existing kernels on the system, but I doubt
that it would be smart enough to actively watch for new kernel installations. 

c)
This has occurred with servers both on the floor(not on racks), and on private
racks. The original was a private rack customer, we put a test server running
rhel4.4 on the very same rack(plugged into the same switch and all) to try and
recreate the issue. We were unable to get the test server to exhibit the same
problems. 

As for connecting to the server directly, this idea was tossed around, but
having the the problem on boxes both on and off the racks, as well as putting a
test server on a rack(containing a server that had the problem) and not showing
any signs of the issue was fairly convincing. 
To be absolutely sure, I will see if I can get another test box to plug directly
into the NIC of one of the buggy servers in the next few days. 
 
Right now we are looking into the possibility of a bad PXE image, however this
is highly unlikely as they have not been changed in quite some time. But at this
point we are simply out of ideas.

To my understanding Kevin L. has contacted Red Hat Support. Unfortunately I do
not believe that Red Hat Support will be able to recreate the issue from the
sysreport.

Comment 6 Joe Orton 2007-04-16 13:01:27 EDT
I noticed that in the config file from the sysreport you have mod_dosevasive20
activated. Can you reproduce the issue without that module?
Comment 7 James Jhurani 2007-04-19 09:59:36 EDT
[root@plesk httpd]# grep -iR dos conf*
conf/magic:#0   leshort         0x76FF          squeezed data (CP/M, DOS)
conf/magic:#0   leshort         0x76FE          crunched data (CP/M, DOS)
[root@plesk httpd]#

webtech@10-33-3-139 ~ > wget http://schoolserver.eduss[snip].com/status/phpinfo.php
--08:54:52--  http://schoolserver.eduss[snip].com/status/phpinfo.php
           => `phpinfo.php'
Resolving schoolserver.eduss[snip].com... 67.15.15x.10
Connecting to schoolserver.eduss[snip].com|67.15.15x.10|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]

    [        <=>                                                               
                      ] 4,162,371      2.44M/s
webtech@10-33-3-139 ~ >    

As you can see the problem still exists without mod_dosevasive20. Also, I
believe using lighthttpd we ruled out any problems caused by an apache module.



Comment 8 Joe Orton 2007-04-19 10:20:39 EDT
To further narrow the source of the corruption can you try two things:

1) Capture a tcpdump running on the server box, simultaneously to one running
remotely, to see whether the packet corruption is evident in the outgoing packets.

2) Check output from httpd process itself:

# service httpd stop
# strace -o /tmp/httpd.trace /usr/sbin/httpd -X
then reproduce the problem, and attach the httpd.trace to this bug report.

Comment 9 Joe Orton 2007-04-19 14:30:26 EDT
Note: please use:

  # strace -s8192 -o /tmp/httpd.trace /usr/sbin/httpd -X

and
 
  # tcpdump -s0 -o /tmp/tcp.dump ...

to ensure the entire packet and system call information is captured.
Comment 10 James Jhurani 2007-04-20 13:17:49 EDT
We also have a ticket open, regarding this issue. In the ticket a different
tcpdump and strace syntax was used. In order to respond to both the ticket and
this bugzilla, I have done both. 

While doing this I noticed something odd, but it makes sense. 

When running apache with strace using -s2048(string size). The page will load
half way, and null terminators, etc. 

But if I use -s8192, as you suggested, the page actually completes. 

Since everyone has their own tcpdump, and strace preference, I tried to keep the
parameters as close to the requested syntax.

209.85.xx.181 == the server
216.12.193.x == me

If you actually want the full ips, let us know, and we will provide them privately. 




Comment 11 James Jhurani 2007-04-20 13:18:49 EDT
Created attachment 153196 [details]
strace string size 2048

client: tcpdump -vvv host 209.85.xx.181 -w tcpdump.2048.client
server: tcpdump -vvv host 216.12.193.x and port 80 -s0 -w tcpdump.2048.server
strace: strace -fxvto http.trace.2048 -s2048 /usr/sbin/httpd -X &
Comment 12 James Jhurani 2007-04-20 13:20:00 EDT
Created attachment 153197 [details]
strace string size 8192

s8192.tar
client: tcpdump -s0 -o tcpdump.8192.client src or dst host 209.85.xx.181 and
src or dst port 80
server: tcpdump -s0 -w tcpdump.8192.server src or dst host 216.12.193.x
strace: strace -s8192 -o http.trace.8192 /usr/sbin/httpd -X &
Comment 13 Joe Orton 2007-04-23 06:27:33 EDT
Thanks.  The fact that the strace options used makes a difference to the
behaviour implies this might be somehow timing related.

The behaviour seen in the -s2048 strace is as follows:

1) first response block is sent with writev()
2) second response block is sent with writev(), returns short
3) httpd does a "poll/writev" loop five times attempting to continue sending the
second block - one byte gets sent each time until the final attempt, which completes
4) subsequent blocks of response are sent without timing out

The packets in the tcpdump look correct exactly up until (4), where only zero
bytes appear.

The checksum mismatch issue may be simply due to TCP checksum offloading.

I can't see any misbehaviour by php/httpd here.  You could try disabling the TCP
offload features for the NIC in question, to see if that makes a difference; e.g:

# ethtool -K ethN tx off sg off tso off
Comment 14 James Jhurani 2007-04-26 08:18:04 EDT
Before ethtool:
webtech@10-33-3-139 ~ > wget http://10.34.1.185/phptest.php
--07:09:53--  http://10.34.1.185/phptest.php
           => `phptest.php.1'
Connecting to 10.34.1.185:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]

    [      <=>                                                                 
                      ] 1,167,427    866.31K/s
webtech@10-33-3-139 ~ >


ran the aforementioned command:
[root@PR-03-actionol html]# ethtool -K eth0 tx off sg off tso off
[root@PR-03-actionol html]# 

webtech@10-33-3-139 ~ > wget http://10.34.1.185/phptest.php
--07:10:33--  http://10.34.1.185/phptest.php
           => `phptest.php.2'
Connecting to 10.34.1.185:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]

    [         <=>                                                              
                      ] 4,268,517      2.12M/s
webtech@10-33-3-139 ~ >  

Unfortunately the problem still exists. 
Comment 15 Joe Orton 2007-04-26 08:40:46 EDT
From the s2048.tar attachment httpd's system call behaviour is correct so this
needs triage from the kernel side, could be some NIC issue.

The writev() call in http.trace.2048:

28245 11:59:04 writev(9, [{"0</td>.../root/bin", 1104}, {"\r\n", 2}], 2) = 1106

corresponds to the first half of frame 23 in tcpdump.2048.server.  Subsequent
writev() calls correspond only to zero bytes in the tcpdump.
Comment 16 James Jhurani 2007-04-26 09:46:15 EDT
We have already tried adding a new(intel) NIC and disabling the onboard. This
did not resolve the situation.

As for the kernel being the problem, this kernel was installed via up2date from
the RHN Satellite. The majority of the servers were using 2.6.9-42.ELsmp. On one
of the 2.6.9-42smp servers, I tried updating the kernel to not only a newer
version, but a uniprocessor(non smp) kernel 2.6.9-42.0.10.EL. The problem still
persisted. 

This was also done previously to rule out an LKM trojan. 

Linux PR-03-actionol 2.6.9-42.0.10.EL #1 Fri Feb 16 17:06:10 EST 2007 i686 i686
i386 GNU/Linux

Are there any tests I can do to give you more information on the kernel? sysrq? 



Comment 17 James Jhurani 2007-05-02 13:20:41 EDT
Are there any updates regarding this issue?
Comment 18 James Jhurani 2007-05-02 14:27:52 EDT
We also just tried reinstalling from CD rather than PXE drive as requested
earlier in the bug report. 

After migrating the data over to the new server(fresh install from CD) the php
issue still occurred. 

Comment 20 James Jhurani 2007-07-04 09:36:51 EDT
We are still waiting on a resolution for this issue. The issue still occurs with
Kernel 2.6.9-55.0.2. 

Note You need to log in before you can comment on or make changes to this bug.