Bug 98542 - iostat -x shows infeasible avgqu-sz results and max util
iostat -x shows infeasible avgqu-sz results and max util
Status: CLOSED ERRATA
Product: Red Hat Enterprise Linux 3
Classification: Red Hat
Component: kernel (Show other bugs)
3.0
i686 Linux
medium Severity medium
: ---
: ---
Assigned To: Larry Woodman
Brian Brock
:
: 117930 (view as bug list)
Depends On:
Blocks: 156321
  Show dependency treegraph
 
Reported: 2003-07-03 09:51 EDT by Michael Wright
Modified: 2007-11-30 17:06 EST (History)
8 users (show)

See Also:
Fixed In Version: RHSA-2005-663
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2005-09-28 10:18:12 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
patch to correct aveq/util computation in iostat (1.32 KB, patch)
2004-08-25 16:32 EDT, Neil Horman
no flags Details | Diff

  None (edit)
Description Michael Wright 2003-07-03 09:51:54 EDT
Description of problem:

Running "iostat -x /dev/hda /dev/hdb /dev/sda /dev/sdb 1 9999" shows avgqu-sz 
as being 42949542.96 and %util as being 100.00, almost constantly... even when 
the disks are under virtually no load.

Occasionally the avgqu-sz will drop to something low like 269.20 or the %util 
drop to 96.00.. both of these do not necessarily happen at the same time and 
both seem to be associated with INCREASED load on the disk! I've never seen %
util drop below 93.00 though.

This behaviour is shown for both /dev/hda and /dev/hdb, but not for /dev/sda 
or /dev/sdb.

If I change the sample time to 4 seconds, instead of one, by running "iostat -
x /dev/hda /dev/hdb /dev/sda /dev/sdb 4 9999" something very odd happens, the 
avgqu-sz values drop by a factor of 4 (so most readings are now 10737288.24 or 
there abouts). The %util is still as high as ever. A sample time of 8 seconds 
shows avgqu-sz figures 8 times less than a sample time of 1 second, etc...

I will attach shell cut-n-pastes for iostat output for sample times of 1, 2 and 
3 seconds.

42949542.96 is recognisable as being 2^32 of course, which suggests to me that 
maybe some system call somewhere is returning a -1 where a real value is 
expected.. then treated as an unsigned integer, if iostat divided the number 
obtained by the number of seconds it was sampling for, to get a per second 
reading, it could account for the other values... and I can conceive that the 
high %util is culculated partly from the avgqu-sz, which would explain that 
too.... but these are all theories of someone who really knows nothing.

I can reproduce this every time of course. My other (redhat 7.2) system does 
not have the problem (it has only a single drive, /dev/hda1).

The very first set of results returned are always quite different, but because 
they are immediate, and not after any sample time, I don't really know what 
they mean!

Version-Release number of selected component (if applicable):

sysstat-4.0.7-3

How reproducible:

Every time.

Steps to Reproduce:

1. Run "iostat -x /dev/hda 1 9999"
2. Observe "avgqu-sz" values (after first result set).

    
Actual results:

I see the following output:

I see the following output:

avg-cpu:  %user   %nice    %sys   %idle
          44.81    0.00    4.47   50.73

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda    34.58  14.05 56.24 26.70  726.56  326.47   363.28   163.23    
12.70    14.78   29.65  12.02  99.68

avg-cpu:  %user   %nice    %sys   %idle
           1.00    0.00    2.00   97.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00 
42949562.96    0.00   0.00 100.00

avg-cpu:  %user   %nice    %sys   %idle
           0.00    0.00    1.00   99.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00 
42949562.96    0.00   0.00 100.00

avg-cpu:  %user   %nice    %sys   %idle
           2.00    0.00    1.00   97.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00 
42949562.96    0.00   0.00 100.00

Expected results:

I would expect the avgqu-sz and %util readings to be low.

Additional info:

My affected system has 4 disks:

  /dev/hda    60Gb IDE
  /dev/hdb    160Gb IDE
  /dev/sda    37Gb SCSI
  /dev/sdb    37Gb SCSI

Abit BE7-S mobo with a 2.5GHz P4 hyperthreading CPU, 1Gb RAM.
Onboard SATA unused and disabled in BIOS.
Comment 1 Michael Wright 2003-07-03 09:55:09 EDT
Output of iostat with sample time 1 second:

[user@host bin]$ iostat -x /dev/hda /dev/hdb /dev/sda /dev/sdb 1 9999
Linux 2.4.20-8 (radiated.local.bogusmove.com)   03/07/03

avg-cpu:  %user   %nice    %sys   %idle
          42.46    0.00    4.38   53.15

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda    35.03  13.32 54.12 25.58  713.16  311.67   356.58   155.83    
12.86     8.80   30.64  12.51  99.70
/dev/hdb    58.71  24.13  9.49  4.44  545.50  228.70   272.75   114.35    
55.57     1.90   44.01  71.74  99.95
/dev/sda   144.75  39.06 11.25  1.52 1247.96  324.57   623.98   162.29   
123.21     0.77    6.03   3.37   4.31
/dev/sdb   211.29  37.77 13.17  1.24 1795.65  312.09   897.83   156.05   
146.27     0.77    5.37   3.06   4.42

avg-cpu:  %user   %nice    %sys   %idle
           1.00    0.00    3.00   96.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda     0.00   8.00  0.00  2.00    0.00   80.00     0.00    40.00    40.00 
42949563.06    5.00 500.00 100.00
/dev/hdb   692.00 1214.00 113.00 204.00 6440.00 11616.00  3220.00  5808.00    
56.96    79.00   46.72   3.15 100.00
/dev/sda     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
/dev/sdb    68.00   0.00  5.00  0.00  584.00    0.00   292.00     0.00   
116.80     0.10    2.00   2.00   1.00

avg-cpu:  %user   %nice    %sys   %idle
           0.00    0.00    4.00   96.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00 
42949562.96    0.00   0.00 100.00
/dev/hdb   588.00 809.00 99.00 155.00 5496.00 7440.00  2748.00  3720.00    
50.93    94.50  112.40   3.90  99.00
/dev/sda     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
/dev/sdb    66.00   0.00 10.00  0.00  608.00    0.00   304.00     0.00    
60.80     0.30    3.00   3.00   3.00

avg-cpu:  %user   %nice    %sys   %idle
           0.00    0.00    0.00  100.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00 
42949562.96    0.00   0.00 100.00
/dev/hdb   1102.00   0.00 187.00  0.00 10312.00    0.00  5156.00     0.00    
55.14 42949555.26    6.63   5.35 100.00
/dev/sda     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
/dev/sdb   162.00   0.00 16.00  0.00 1424.00    0.00   712.00     0.00    
89.00     0.60    3.75   2.50   4.00

avg-cpu:  %user   %nice    %sys   %idle
           0.00    0.00    0.00  100.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00 
42949562.96    0.00   0.00 100.00
/dev/hdb   1150.00   0.00 199.00  0.00 10792.00    0.00  5396.00     0.00    
54.23 42949554.76    5.93   5.03 100.00
/dev/sda     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
/dev/sdb   128.00   0.00  6.00  0.00 1072.00    0.00   536.00     0.00   
178.67     0.10    1.67   1.67   1.00


[user@host bin]$
Comment 2 Michael Wright 2003-07-03 09:58:16 EDT
Output of iostat with sample time 2 seconds:

[user@host bin]$ iostat -x /dev/hda /dev/hdb /dev/sda /dev/sdb 2 9999
Linux 2.4.20-8 (host)   03/07/03

avg-cpu:  %user   %nice    %sys   %idle
          42.02    0.00    4.37   53.61

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda    34.78  13.18 53.69 25.53  707.67  310.13   353.84   155.06    
12.85     8.17   31.45  12.59  99.70
/dev/hdb    67.52  27.88 10.89  5.15  627.21  264.38   313.61   132.19    
55.61     1.45   44.79  62.34  99.95
/dev/sda   143.03  38.59 11.11  1.50 1233.10  320.71   616.55   160.35   
123.21     0.76    6.03   3.37   4.25
/dev/sdb   212.68  37.32 13.19  1.23 1806.94  308.38   903.47   154.19   
146.73     0.77    5.33   3.05   4.40

avg-cpu:  %user   %nice    %sys   %idle
          21.50    0.00    1.50   77.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda    19.50   6.50 29.50 78.50  392.00  680.00   196.00   340.00     
9.93   112.05  202.55   9.26 100.00
/dev/hdb   415.00 262.50 61.00 44.00 3800.00 2452.00  1900.00  1226.00    59.54 
21474774.83   64.38   9.52 100.00
/dev/sda     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
/dev/sdb    59.50   0.00  5.00  0.00  516.00    0.00   258.00     0.00   
103.20     0.10    2.00   2.00   1.00

avg-cpu:  %user   %nice    %sys   %idle
           8.50    0.00    2.50   89.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda   109.50   0.00 42.00  0.00 1200.00    0.00   600.00     0.00    28.57 
21474735.63   30.36  23.81 100.00
/dev/hdb   1033.50   0.00 157.00  0.00 9524.00    0.00  4762.00     0.00    
60.66 21474719.23    8.76   6.37 100.00
/dev/sda     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
/dev/sdb   1108.50   0.00 39.50  0.00 9184.00    0.00  4592.00     0.00   
232.51     0.95    2.41   2.41   9.50

avg-cpu:  %user   %nice    %sys   %idle
           1.00    0.00    2.00   97.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda     1.00  25.50  7.50 36.00   76.00  612.00    38.00   306.00    15.82 
21474790.78   75.75  22.99 100.00
/dev/hdb   1214.50 414.50 189.50 40.00 11232.00 3872.00  5616.00  1936.00    
65.81 21474775.03   18.17   4.36 100.00
/dev/sda     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
/dev/sdb   155.00   0.00  8.50  0.00 1308.00    0.00   654.00     0.00   
153.88     0.25    2.94   2.94   2.50


[user@host bin]$
Comment 3 Michael Wright 2003-07-03 09:59:44 EDT
[user@host bin]$ iostat -x /dev/hda /dev/hdb /dev/sda /dev/sdb 3 9999
Linux 2.4.20-8 host)   03/07/03

avg-cpu:  %user   %nice    %sys   %idle
          41.63    0.00    4.35   54.01

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda    34.51  13.05 53.39 25.57  703.15  309.46   351.58   154.73    
12.82     7.77   32.50  12.63  99.70
/dev/hdb    75.81  31.29 12.26  5.82  704.48  297.01   352.24   148.50    
55.39     0.86   44.91  55.28  99.95
/dev/sda   141.27  38.12 10.98  1.48 1217.93  316.76   608.96   158.38   
123.21     0.75    6.03   3.37   4.20
/dev/sdb   214.23  36.86 13.19  1.21 1819.31  304.58   909.66   152.29   
147.51     0.76    5.30   3.04   4.38

avg-cpu:  %user   %nice    %sys   %idle
           0.67    0.00    0.67   98.67

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda     0.00   0.00  0.33  0.00    2.67    0.00     1.33     0.00     8.00 
14316447.69   10.00 3000.00 100.00
/dev/hdb   1036.00  20.67 189.67  6.00 9802.67  218.67  4901.33   109.33    
51.22 14316439.75    6.17   5.11 100.00
/dev/sda     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
/dev/sdb   739.67   0.00 26.00  0.00 6125.33    0.00  3062.67     0.00   
235.59     0.53    2.05   2.05   5.33

avg-cpu:  %user   %nice    %sys   %idle
           0.33    0.00    1.33   98.33

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda     0.00   2.67  0.00  0.67    0.00   26.67     0.00    13.33    40.00 
14316447.72   10.00 1500.00 100.00
/dev/hdb   677.33 635.00 114.00 133.00 6330.67 6144.00  3165.33  3072.00    
50.50    78.10   84.24   4.04  99.67
/dev/sda     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
/dev/sdb    78.67   0.00  3.00  0.00  653.33    0.00   326.67     0.00   
217.78     0.03    1.11   1.11   0.33

avg-cpu:  %user   %nice    %sys   %idle
           0.67    0.00    5.00   94.33

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda     0.00   0.33  0.00  0.67    0.00    8.00     0.00     4.00    12.00 
14316447.69    5.00 1500.00 100.00
/dev/hdb   865.33 572.67 131.00 95.67 7970.67 5402.67  3985.33  2701.33    
59.00 14316542.25   45.15   4.41 100.00
/dev/sda     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
/dev/sdb   734.00   0.00 51.00  0.00 6280.00    0.00  3140.00     0.00   
123.14     6.10   11.96   2.42  12.33


[user@host bin]$
Comment 4 Michael Wright 2003-07-03 10:01:12 EDT
Note that /dev/hdb was under load at the time these samples were taken, 
but /dev/hda was not, I believe, under any significant load.
Comment 5 Michael Wright 2003-07-03 18:07:52 EDT
Hmm, new and interesting stats :) Under load, I'm now
seeing %util as low as 62.59 and as high as 128.37!!!
Now that's impressive.. a disk that is 128% utilised :)

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda   386.56   6.32 177.47 11.86 4569.17  145.45  2284.58    72.73    
24.90   108.06  120.96   6.10 115.42
/dev/hdb     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00 
16976003.74    0.00   0.00 116.60
/dev/sda    15.81  44.66  2.77  7.11  148.62  414.23    74.31   207.11    
56.96     0.08    0.80   0.80   0.79
/dev/sdb     0.40  25.30 66.80 100.79  540.71 1008.70   270.36   504.35     
9.25    53.44   31.86   3.09  51.78

avg-cpu:  %user   %nice    %sys   %idle
           9.52    0.00    7.48   82.99

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda   157.82  29.25 117.69  6.80 2160.54  288.44  1080.27   144.22    
19.67   115.24  150.27   5.03  62.59
/dev/hdb     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00 
29217383.24    0.00   0.00  62.59
/dev/sda     0.00   0.68  0.00  2.72    0.00   27.21     0.00    13.61    
10.00     0.00    0.00   0.00   0.00
/dev/sdb     0.00  12.24 112.93  8.16  903.40  163.27   451.70    81.63     
8.81     6.19    5.11   4.27  51.70

avg-cpu:  %user   %nice    %sys   %idle
          47.52    0.00    6.44   46.04

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda   428.71  10.40 158.42 24.26 4685.15  277.23  2342.57   138.61    
27.17   120.84  135.80   6.12 111.88
/dev/hdb     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00 
21262067.60    0.00   0.00 112.87
/dev/sda     6.93  14.36  1.98  1.49   71.29  126.73    35.64    63.37    
57.14     0.00    0.00   0.00   0.00
/dev/sdb     0.00   2.48 98.02 62.38  780.20  534.65   390.10   267.33     
8.20     6.88    4.32   3.61  57.92

avg-cpu:  %user   %nice    %sys   %idle
           5.79    0.00    5.02   89.19

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda   324.71  16.22 168.34  8.88 3987.64  206.95  1993.82   103.47    
23.67   164.05  148.63   5.47  96.91
/dev/hdb     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00 
16582759.33    0.00   0.00  96.91
/dev/sda    18.53  25.87  3.09  4.63  172.97  244.02    86.49   122.01    
54.00     0.08    1.00   1.00   0.77
/dev/sdb     0.00  27.41 100.00 132.82  800.00 1269.50   400.00   634.75     
8.89   106.83   45.89   3.40  79.15

avg-cpu:  %user   %nice    %sys   %idle
          14.18    0.00    7.80   78.01

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda   342.55   1.42 143.26 17.73 3795.74  147.52  1897.87    73.76    
24.49   150.43  167.09   5.95  95.74
/dev/hdb     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00 
30460636.50    0.00   0.00  95.74
/dev/sda     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
/dev/sdb     0.00   2.84 131.91 30.50 1060.99  266.67   530.50   133.33     
8.17     6.95    4.28   3.84  62.41

avg-cpu:  %user   %nice    %sys   %idle
          13.54    0.00    6.11   80.35

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda   440.17   0.00 205.68  1.75 5156.33   10.48  2578.17     5.24    
24.91   109.61  110.44   5.05 104.80
/dev/hdb     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00 
18755177.76    0.00   0.00 106.11
/dev/sda     7.42   0.44  3.06  1.31   83.84   13.97    41.92     6.99    
22.40     0.04    1.00   1.00   0.44
/dev/sdb     0.00   9.17 121.83 105.68  974.67  918.78   487.34   459.39     
8.32    10.92    4.78   3.90  88.65

avg-cpu:  %user   %nice    %sys   %idle
           5.11    0.00    5.68   89.20

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda   387.50   6.25 127.27 10.23 4127.27  131.82  2063.64    65.91    
30.98    17.90   84.34   6.40  88.07
/dev/hdb     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00 
24403106.57    0.00   0.00  89.77
/dev/sda     0.00   0.57  1.14  3.41    9.09   31.82     4.55    15.91     
9.00     0.06    1.25   1.25   0.57
/dev/sdb     0.00  39.20 65.91 131.82  522.73 1368.18   261.36   684.09     
9.56   184.09   93.13   3.22  63.64

avg-cpu:  %user   %nice    %sys   %idle
          13.89    0.00    6.94   79.17

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda   435.19   0.00 162.96  0.46 4774.07    3.70  2387.04     1.85    
29.24 19884081.23   53.12   5.47  89.35
/dev/hdb     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00 
19883975.44    0.00   0.00 101.85
/dev/sda     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
/dev/sdb     0.00   2.31 87.50 38.89  700.00  329.63   350.00   164.81     
8.15     5.09    4.03   3.66  46.30

avg-cpu:  %user   %nice    %sys   %idle
           6.74    0.00    4.15   89.12

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda   399.48   5.18 168.39  2.07 4559.59   58.03  2279.79    29.02    
27.09    62.85  106.81   5.99 102.07
/dev/hdb     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00 
22253573.76    0.00   0.00 109.84
/dev/sda     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
/dev/sdb     0.52  40.41 101.55 167.36  820.73 1662.18   410.36   831.09     
9.23   250.88   93.29   3.03  81.35

avg-cpu:  %user   %nice    %sys   %idle
          20.44    0.00    7.11   72.44

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda   317.33  14.67 165.33 18.22 3879.11  266.67  1939.56   133.33    
22.59    92.31  115.86   5.93 108.89
/dev/hdb     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00 
19088597.36    0.00   0.00 112.44
/dev/sda     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
/dev/sdb     0.00   2.67 164.00 49.78 1312.00  419.56   656.00   209.78     
8.10     9.82    4.57   3.91  83.56

avg-cpu:  %user   %nice    %sys   %idle
          45.03    0.00    7.85   47.12

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda   251.31  10.47 117.80  8.90 2923.56  150.79  1461.78    75.39    
24.26    45.60  103.02   5.58  70.68
/dev/hdb     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00 
22486645.16    0.00   0.00  72.77
/dev/sda     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
/dev/sdb     0.00  11.52 92.67 59.69  741.36  569.63   370.68   284.82     
8.60     6.60    4.36   3.75  57.07

avg-cpu:  %user   %nice    %sys   %idle
          14.90    0.00    5.77   79.33

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda   374.04  21.63 212.50 16.83 4730.77  323.08  2365.38   161.54    
22.04   130.96  112.43   5.35 122.60
/dev/hdb     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00 
20648714.36    0.00   0.00 128.37
/dev/sda     0.00  23.08  3.37  2.40   26.92  203.85    13.46   101.92    
40.00     0.00    0.00   0.00   0.00
/dev/sdb     0.00  44.71 148.56 175.00 1188.46 1769.23   594.23   884.62     
9.14   173.51   53.61   3.15 101.92

avg-cpu:  %user   %nice    %sys   %idle
          21.50    0.00    8.00   70.50

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda   181.50   0.00 118.00  7.00 2380.00   44.00  1190.00    22.00    
19.39    92.05  146.20   5.88  73.50
/dev/hdb     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00 
21474740.28    0.00   0.00  74.00
/dev/sda     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
/dev/sdb     0.00   1.50 85.00 30.00  676.00  240.00   338.00   120.00     
7.97     5.55    4.87   3.91  45.00

avg-cpu:  %user   %nice    %sys   %idle
          18.08    0.00    4.80   77.12

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda   313.65  14.39 169.74 13.28 3837.64  218.45  1918.82   109.23    
22.16    64.76  102.30   5.56 101.85
/dev/hdb     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00 
15848449.17    0.00   0.00 106.64
/dev/sda     0.00   2.21  1.48  2.58   11.81   38.38     5.90    19.19    
12.36     0.00    0.00   0.00   0.00
/dev/sdb     0.00  51.66 112.55 171.22  903.32 1783.03   451.66   891.51     
9.47   260.30   91.72   3.20  90.77

avg-cpu:  %user   %nice    %sys   %idle
          15.76    0.00    5.45   78.79

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda   145.45   0.00 101.82 13.33 2050.91  106.67  1025.45    53.33    
18.74 26030095.07   66.79   6.58  75.76
/dev/hdb     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00 
26030000.04    0.00   0.00  80.61
/dev/sda     0.00   1.21  1.21  0.61    9.70   14.55     4.85     7.27    
13.33     0.00    0.00   0.00   0.00
/dev/sdb     0.00   1.21 136.97 20.00 1095.76  169.70   547.88    84.85     
8.06     7.94    5.06   4.21  66.06

avg-cpu:  %user   %nice    %sys   %idle
           9.30    0.00    5.58   85.12

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda   440.47   0.00 195.35  3.72 5026.98   29.77  2513.49    14.88    
25.40 19976578.77   54.95   5.19 103.26
/dev/hdb     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00 
19976449.38    0.00   0.00 109.77
/dev/sda     0.00  14.42  0.00  2.33    0.00  133.95     0.00    66.98    
57.60     0.00    0.00   0.00   0.00
/dev/sdb     0.00  15.35 133.02 120.47 1064.19 1086.51   532.09   543.26     
8.48    14.93    5.82   3.58  90.70

avg-cpu:  %user   %nice    %sys   %idle
          14.14    0.00    5.24   80.63

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda   241.88  10.47 112.57  9.95 2848.17  167.54  1424.08    83.77    
24.62 22486705.84   50.34   6.50  79.58
/dev/hdb     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00 
22486624.06    0.00   0.00  89.01
/dev/sda     0.00   3.14  1.05  3.14    8.38   50.26     4.19    25.13    
14.00     0.00    0.00   0.00   0.00
/dev/sdb     0.00  42.93 105.76 136.13  846.07 1432.46   423.04   716.23     
9.42   241.73  100.02   3.25  78.53

avg-cpu:  %user   %nice    %sys   %idle
          25.85    0.00    7.32   66.83

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda   218.54   3.41 173.17  4.88 3118.05   62.44  1559.02    31.22    
17.86     5.95   69.10   5.67 100.98
/dev/hdb     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00 
20950924.27    0.00   0.00 104.39
/dev/sda     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     
0.00     0.00    0.00   0.00   0.00
/dev/sdb     0.00   4.39 144.39 46.34 1155.12  405.85   577.56   202.93     
8.18     9.37    4.91   3.99  76.10

avg-cpu:  %user   %nice    %sys   %idle
          10.50    0.00    4.00   85.50

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz 
avgqu-sz   await  svctm  %util
/dev/hda   226.00  28.00 117.00 17.50 2768.00  372.00  1384.00   186.00    
23.35 21474817.48   61.23   6.91  93.00
/dev/hdb     0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00 
21474712.33    0.00   0.00  95.50
/dev/sda    12.50   2.50  3.00  3.00  124.00   44.00    62.00    22.00    
28.00     0.00    0.00   0.00   0.00
/dev/sdb     0.00  44.50 65.00 217.00  520.00 2124.00   260.00  1062.00     
9.38   320.45  105.25   3.05  86.00
Comment 6 Don Harris 2004-03-10 08:33:11 EST
I have seen this behavior on our RHEL 3.0 system as well.  It's very concerning to me, 
mainly because I think on some level this is effecting the hard drive performance.  Judging 
solely by the "little yellow" hard drive activity light (that once flickered when it was active) 
it is now solid amber, with no inactivity.  My concern is that the problem is continually 
spinning the disk drive(s).
As previously stated, I can reproduce every time, as well as what Michael Wright noticed, 
the avgqu-sz does seem to be related to the 'duration' of iostat.  I feel that the problem is 
deeper than just a problem with iostat.
Comment 7 Dean Wilson 2004-03-23 08:18:30 EST
I can reproduce this behaviour on a number of Dell Poweredge machines
(HyperThreading) running RH versions 7.1, 7.2, 7.3, 8.0 and 9.0 with
stock kernels.
                                                                     
          
I see the exact same results as the other commenters.
Comment 8 Charlie Bennett 2004-04-07 13:55:55 EDT
*** Bug 117930 has been marked as a duplicate of this bug. ***
Comment 9 Bastien Nocera 2004-07-21 07:05:09 EDT
As RHL9 is dead, and the same thing happens on RHEL3, reassigning.
Comment 10 Bastien Nocera 2004-07-22 10:53:56 EDT
The version of sysstat affected is 4.0.7-4.EL3.2
Comment 11 Bastien Nocera 2004-08-02 10:56:38 EDT
The problem only happens with IDE disks, not with SCSI devices.
Comment 12 Michael Wright 2004-08-04 05:42:22 EDT
That (SCSI disks are not affected) was my experience also. Note too 
that not all systems or all IDE disks are affected as well, I have a 
RedHat 7.2 system with a single IDE disk which does not exhibit the 
problem.
Comment 13 Neil Horman 2004-08-25 16:32:00 EDT
Created attachment 103097 [details]
patch to correct aveq/util computation in iostat

i'm not 100% sure of this, but I think that /proc/partitions (which is used in
the beta for iostat) is miscomputing aveq-sz and utilization.  The kernel
tracks these values in proc/partitions as differences between idle times, and
so I don't think an extra difference computation in iostat is needed.  Also the
utilization values in /proc/partitions are given in ticks, so I believe that
util computation was off by a factor of 10.  This patch (applied to the U3 beta
of sysstat) fixes those problems.
Comment 15 Johnray Fuller 2004-09-02 23:58:40 EDT
The from Comment #13 appears to cause %util numbers that are too low.

Here is iostat -x 1 on an unloaded disk. The avgqu-sz is too high. The 
svctm is too high (intermitantly), and the %util seems low.

avg-cpu:  %user   %nice    %sys   %idle
           4.95    0.00    0.99   94.06
 
Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s
avgrq-sz avgqu-sz   await  svctm  %util
/dev/hda     0.00   8.91  0.00  3.96    0.00  102.97     0.00    51.49
   26.00 348700.83    0.00 252.50   0.10

Here is the same iostat -x 1 command with a loaded disk (cat /dev/hda
> /dev/null):

avg-cpu:  %user   %nice    %sys   %idle
         100.00    0.00    0.00    0.00
 
Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s
avgrq-sz avgqu-sz   await  svctm  %util
/dev/hda   18614.29 814.29 1114.29 71.43 48228.57 7085.71 24114.29 
3542.86    46.65 3772660.57   90.84  12.05   1.43

Again, the %util is low as the disk is heavily loaded, the avgqu-sz is
really up there, and the svctm seems pretty reasonable.
Comment 17 Charlie Bennett 2004-10-04 15:57:08 EDT

*** This bug has been marked as a duplicate of 134579 ***
Comment 18 Ernie Petrides 2005-09-08 19:19:16 EDT
Fixing component and assignee.
Comment 19 Ernie Petrides 2005-09-08 19:22:21 EDT
Propagating acks from bug 134579.

*** This bug has been marked as a duplicate of 134579 ***
Comment 20 Red Hat Bugzilla 2005-09-28 10:18:12 EDT
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHSA-2005-663.html

Note You need to log in before you can comment on or make changes to this bug.