Bug 79726 - strange system load on rh 7.2 web servers
strange system load on rh 7.2 web servers
Status: CLOSED WORKSFORME
Product: Red Hat Enterprise Linux 2.1
Classification: Red Hat
Component: kernel (Show other bugs)
2.1
i686 Linux
medium Severity medium
: ---
: ---
Assigned To: Larry Woodman
Brian Brock
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2002-12-16 04:28 EST by Tobias Meier
Modified: 2007-11-30 17:06 EST (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2005-09-28 07:40:59 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
here is the kernel debug stream (201.99 KB, text/plain)
2002-12-18 05:04 EST, Tobias Meier
no flags Details
new debug stream (98.81 KB, text/plain)
2002-12-18 10:06 EST, Tobias Meier
no flags Details
debug output advanced server (147.86 KB, application/octet-stream)
2003-01-09 08:10 EST, Tobias Meier
no flags Details
the same file again / without auto-detect content type (147.86 KB, text/plain)
2003-01-09 08:15 EST, Tobias Meier
no flags Details
lspci output (8.35 KB, text/plain)
2003-01-09 09:00 EST, Tobias Meier
no flags Details
/ets/fstab (1.04 KB, text/plain)
2003-01-09 09:04 EST, Tobias Meier
no flags Details
interesting top output --> kswapd (2.09 KB, text/plain)
2003-01-09 10:38 EST, Tobias Meier
no flags Details
new rh 7.2 kernel debug stream (543.30 KB, text/plain)
2003-01-13 16:56 EST, Tobias Meier
no flags Details
top vmstat and ps output while the load is high (30.88 KB, text/plain)
2003-01-14 13:14 EST, Tobias Meier
no flags Details

  None (edit)
Description Tobias Meier 2002-12-16 04:28:51 EST
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0.1) Gecko/20021003

Description of problem:
hi,
we use FSC Primary P250 boxes with 2 * 2,4 ghz xeon processors and a adaptec
2100S raid controller. the os is redhat 7.2 and a 2.4.18-18-7.xsmp kernel.
we installed a squid and an apache server. the normal system load is between 0.1
and 0.5 (/proc/loadavg). but sometimes the load goes up to 4-6, and we don't
know why. the cpus are 98% idle and there is no io traffic.
after 2 or 3 hours the load goes back down to 0.1. the same problem appears with
a uniprocessor kernel.
as soon as we stop the squid , the load goes back down to a normal value, but
after restarting the squid, the problem appears again. 

interesting: starting a bonnie++ while the load is up, the load goes up to
10-12, killing the bonnie process 2 minutes later results in the load going back
down to a normal value (0.2) and stays down.

sometimes the top output and the sar tool shows wrong values. i saw an idle time
of 234567.98 for example or a cpu usage of 234567%.
the other values report that the system feels boring, and the squid response
times looks good :-)

we used 4 different setups: 4 boxes with smp kernel and redhat squid, 3 with up
kernel an redhat squid, 4 with smp kernel and our own squid with mod_gzip and 2
with up kernel and our squid. on all systems the problem appears.

   tobias

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1.install a fsc xeon box with rh 7.2
2.install a squid and a apache
3.simulate traffic ( 100 requests/s ) 
4.wait 1 or 2 weeks and monitor the load
    

Additional info:
Comment 1 Arjan van de Ven 2002-12-16 04:52:18 EST
if you can enable sysreq ("echo 1 > /proc/sys/kernel/sysrq") then using the
alt-sysrq-t key combination will spew a kernel debug stream to syslog. based on
that it's possible to see why/where the load is so high, please attach such
output here. (but only in the problem scenario; in the "healthy" case it's no use)
Comment 2 Ben LaHaise 2002-12-16 11:31:50 EST
There are two separate bugs here: the obviously incorrect cpu usage (which may
be related to a missed timer tick or incorrect time accounting in the kernel),
and the high load triggering process death (likely a vm issue).
Comment 3 Tobias Meier 2002-12-18 05:04:02 EST
Created attachment 88791 [details]
here is the kernel debug stream
Comment 4 Arjan van de Ven 2002-12-18 05:19:08 EST
looks like it got stuck on NFS :(
Comment 5 Tobias Meier 2002-12-18 05:32:32 EST
we had the same problems without nfs. we can unmount the nfs devices and send
you a new debug stream.
Comment 6 Tobias Meier 2002-12-18 10:06:40 EST
Created attachment 88795 [details]
new debug stream

after umount all nfs shares, stop the nfs services, and rmmod the nfs kernel
modul. the load is still 4.
Comment 7 Tobias Meier 2002-12-18 10:45:18 EST
 fyi: we have the same problems as descriped in bug id: 64984. perhaps our 
load problems may result from this bug.  
 
Comment 8 Tobias Meier 2003-01-09 08:00:57 EST
ok, we have the same problems with redhat advanced server. we solved our nfs
problems and the problem is still here.
Comment 9 Tobias Meier 2003-01-09 08:10:42 EST
Created attachment 89234 [details]
debug output advanced server
Comment 10 Tobias Meier 2003-01-09 08:15:26 EST
Created attachment 89235 [details]
the same file again / without auto-detect content type
Comment 11 Tobias Meier 2003-01-09 09:00:17 EST
Created attachment 89236 [details]
lspci output
Comment 12 Tobias Meier 2003-01-09 09:04:48 EST
Created attachment 89237 [details]
/ets/fstab
Comment 13 Tobias Meier 2003-01-09 09:36:38 EST
is there a way to get the kernel debug stream without pressing the sysrq keys ?
Comment 14 Tobias Meier 2003-01-09 10:38:47 EST
Created attachment 89243 [details]
interesting top output --> kswapd
Comment 15 Tobias Meier 2003-01-13 16:56:27 EST
Created attachment 89341 [details]
new rh 7.2 kernel debug stream
Comment 16 Bastien Nocera 2003-01-14 12:02:28 EST
Hello Tobias,

We would need some more information, related to your finding about kswapd.

We would need the data from:
- readprofile
- top
- vmstat
when the problem occurs. It will give us more depth into the problem, now that
we know that kswapd might be a problem.

Cheers
Comment 17 Tobias Meier 2003-01-14 13:14:53 EST
Created attachment 89356 [details]
top vmstat and ps output while the load is high
Comment 18 Larry Woodman 2005-09-28 07:40:59 EDT
Let me know if this is still a problem.

Larry Woodman

Note You need to log in before you can comment on or make changes to this bug.