From Bugzilla Helper: User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; Q312461; .NET CLR 1.0.3705) Description of problem: When running 'top' on an idle RH7.3 system, top consumes an exceptionally large amount of cpu. vmstat shows 98% cpu idle. Version-Release number of selected component (if applicable): How reproducible: Always Steps to Reproduce: 1. vmstat 5 2. top 3. # vmstat 5 procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 0 0 0 84 190188 115568 1532592 0 0 0 35 209 127 1 1 98 2 0 0 84 187932 115568 1532592 0 0 0 30 238 164 4 1 95 0 0 0 84 190152 115568 1532592 0 0 0 24 299 216 9 3 89 0 0 0 84 190144 115568 1532592 0 0 0 45 253 178 1 1 98 # top 3:00pm up 15 days, 10 min, 1 user, load average: 0.48, 0.34, 0.82 160 processes: 159 sleeping, 1 running, 0 zombie, 0 stopped CPU0 states: 0.4% user, 54.1% system, 0.0% nice, 45.5% idle CPU1 states: 0.6% user, 0.3% system, 0.0% nice, 99.1% idle Mem: 3223428K av, 3033456K used, 189972K free, 0K shrd, 115528K buff Swap: 2097112K av, 84K used, 2097028K free 1532592K cached PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND 22321 root 14 0 1120 1120 844 R 53.9 0.0 0:40 top 12205 mysql 9 0 1179M 1.2G 1784 S 0.5 37.4 4:47 mysqld 12216 mysql 9 0 1179M 1.2G 1784 S 0.2 37.4 4:49 mysqld 12293 mysql 9 0 1179M 1.2G 1784 S 0.2 37.4 5:05 mysqld 12218 mysql 9 0 1179M 1.2G 1784 S 0.1 37.4 4:48 mysqld 12245 mysql 9 0 1179M 1.2G 1784 S 0.1 37.4 5:04 mysqld 12250 mysql 9 0 1179M 1.2G 1784 S 0.1 37.4 4:54 mysqld 12271 mysql 9 0 1179M 1.2G 1784 S 0.1 37.4 5:00 mysqld 1 root 9 0 404 384 368 S 0.0 0.0 0:15 init 2 root 9 0 0 0 0 SW 0.0 0.0 0:00 keventd 3 root 19 19 0 0 0 SWN 0.0 0.0 0:00 ksoftirqd_CPU0 4 root 19 19 0 0 0 SWN 0.0 0.0 0:00 ksoftirqd_CPU1 5 root 9 0 0 0 0 SW 0.0 0.0 0:02 kswapd 6 root 9 0 0 0 0 SW 0.0 0.0 0:00 bdflush 7 root 9 0 0 0 0 SW 0.0 0.0 0:15 kupdated 8 root 9 0 0 0 0 SW 0.0 0.0 0:00 kjournald 132 root 9 0 0 0 0 SW 0.0 0.0 0:00 kjournald 133 root 9 0 0 0 0 SW 0.0 0.0 1:42 kjournald 134 root 9 0 0 0 0 SW 0.0 0.0 0:00 kjournald 135 root 9 0 0 0 0 SW 0.0 0.0 0:33 kjournald 136 root 9 0 0 0 0 SW 0.0 0.0 1:09 kjournald 137 root 9 0 0 0 0 SW 0.0 0.0 0:01 kjournald 491 root 9 0 540 540 452 S 0.0 0.0 0:01 syslogd 496 root 9 0 440 432 380 S 0.0 0.0 0:00 klogd 584 root 8 0 612 612 532 S 0.0 0.0 0:01 crond 669 daemon 9 0 516 504 452 S 0.0 0.0 0:00 atd Additional info: The system is a standard RH 7.3 (no-x) install running on a Dual Pentium 3 server with 3Gb of RAM. The kernel running is vanilla 2.4.18 with no patches and compiled with 4GB Highmem support.
Do you only see this on highmem system?
This symptom was not seen previously before the current server setup however we made several changes on the system. Previously the system had 1GB of physical RAM and vanilla 2.4.18 kernel and we locked the mysqld process to use 256MB or so of this. However after the upgrade of memory to 3GB with the required 4GB highmem kernel setting and removing the memory limit on mysql the symptom appeared. We have tried the same highmem kernel on other RH server running mysql, these systems have 512MB - 1GB of RAM without the same problem with top. Not understanding how top or highmem works exactly, but to me it looks like its taking a lot of cpu to read the /proc info for each 1GB+ mysql thread.
Ok. I talked to the kernel people. There is a patch that fixes this in later versions of the kernel (not the one we ship in 7.3 though, although AS has it).