Description of problem: Running "iostat -x /dev/hda /dev/hdb /dev/sda /dev/sdb 1 9999" shows avgqu-sz as being 42949542.96 and %util as being 100.00, almost constantly... even when the disks are under virtually no load. Occasionally the avgqu-sz will drop to something low like 269.20 or the %util drop to 96.00.. both of these do not necessarily happen at the same time and both seem to be associated with INCREASED load on the disk! I've never seen % util drop below 93.00 though. This behaviour is shown for both /dev/hda and /dev/hdb, but not for /dev/sda or /dev/sdb. If I change the sample time to 4 seconds, instead of one, by running "iostat - x /dev/hda /dev/hdb /dev/sda /dev/sdb 4 9999" something very odd happens, the avgqu-sz values drop by a factor of 4 (so most readings are now 10737288.24 or there abouts). The %util is still as high as ever. A sample time of 8 seconds shows avgqu-sz figures 8 times less than a sample time of 1 second, etc... I will attach shell cut-n-pastes for iostat output for sample times of 1, 2 and 3 seconds. 42949542.96 is recognisable as being 2^32 of course, which suggests to me that maybe some system call somewhere is returning a -1 where a real value is expected.. then treated as an unsigned integer, if iostat divided the number obtained by the number of seconds it was sampling for, to get a per second reading, it could account for the other values... and I can conceive that the high %util is culculated partly from the avgqu-sz, which would explain that too.... but these are all theories of someone who really knows nothing. I can reproduce this every time of course. My other (redhat 7.2) system does not have the problem (it has only a single drive, /dev/hda1). The very first set of results returned are always quite different, but because they are immediate, and not after any sample time, I don't really know what they mean! Version-Release number of selected component (if applicable): sysstat-4.0.7-3 How reproducible: Every time. Steps to Reproduce: 1. Run "iostat -x /dev/hda 1 9999" 2. Observe "avgqu-sz" values (after first result set). Actual results: I see the following output: I see the following output: avg-cpu: %user %nice %sys %idle 44.81 0.00 4.47 50.73 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 34.58 14.05 56.24 26.70 726.56 326.47 363.28 163.23 12.70 14.78 29.65 12.02 99.68 avg-cpu: %user %nice %sys %idle 1.00 0.00 2.00 97.00 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 42949562.96 0.00 0.00 100.00 avg-cpu: %user %nice %sys %idle 0.00 0.00 1.00 99.00 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 42949562.96 0.00 0.00 100.00 avg-cpu: %user %nice %sys %idle 2.00 0.00 1.00 97.00 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 42949562.96 0.00 0.00 100.00 Expected results: I would expect the avgqu-sz and %util readings to be low. Additional info: My affected system has 4 disks: /dev/hda 60Gb IDE /dev/hdb 160Gb IDE /dev/sda 37Gb SCSI /dev/sdb 37Gb SCSI Abit BE7-S mobo with a 2.5GHz P4 hyperthreading CPU, 1Gb RAM. Onboard SATA unused and disabled in BIOS.
Output of iostat with sample time 1 second: [user@host bin]$ iostat -x /dev/hda /dev/hdb /dev/sda /dev/sdb 1 9999 Linux 2.4.20-8 (radiated.local.bogusmove.com) 03/07/03 avg-cpu: %user %nice %sys %idle 42.46 0.00 4.38 53.15 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 35.03 13.32 54.12 25.58 713.16 311.67 356.58 155.83 12.86 8.80 30.64 12.51 99.70 /dev/hdb 58.71 24.13 9.49 4.44 545.50 228.70 272.75 114.35 55.57 1.90 44.01 71.74 99.95 /dev/sda 144.75 39.06 11.25 1.52 1247.96 324.57 623.98 162.29 123.21 0.77 6.03 3.37 4.31 /dev/sdb 211.29 37.77 13.17 1.24 1795.65 312.09 897.83 156.05 146.27 0.77 5.37 3.06 4.42 avg-cpu: %user %nice %sys %idle 1.00 0.00 3.00 96.00 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 0.00 8.00 0.00 2.00 0.00 80.00 0.00 40.00 40.00 42949563.06 5.00 500.00 100.00 /dev/hdb 692.00 1214.00 113.00 204.00 6440.00 11616.00 3220.00 5808.00 56.96 79.00 46.72 3.15 100.00 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 68.00 0.00 5.00 0.00 584.00 0.00 292.00 0.00 116.80 0.10 2.00 2.00 1.00 avg-cpu: %user %nice %sys %idle 0.00 0.00 4.00 96.00 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 42949562.96 0.00 0.00 100.00 /dev/hdb 588.00 809.00 99.00 155.00 5496.00 7440.00 2748.00 3720.00 50.93 94.50 112.40 3.90 99.00 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 66.00 0.00 10.00 0.00 608.00 0.00 304.00 0.00 60.80 0.30 3.00 3.00 3.00 avg-cpu: %user %nice %sys %idle 0.00 0.00 0.00 100.00 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 42949562.96 0.00 0.00 100.00 /dev/hdb 1102.00 0.00 187.00 0.00 10312.00 0.00 5156.00 0.00 55.14 42949555.26 6.63 5.35 100.00 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 162.00 0.00 16.00 0.00 1424.00 0.00 712.00 0.00 89.00 0.60 3.75 2.50 4.00 avg-cpu: %user %nice %sys %idle 0.00 0.00 0.00 100.00 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 42949562.96 0.00 0.00 100.00 /dev/hdb 1150.00 0.00 199.00 0.00 10792.00 0.00 5396.00 0.00 54.23 42949554.76 5.93 5.03 100.00 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 128.00 0.00 6.00 0.00 1072.00 0.00 536.00 0.00 178.67 0.10 1.67 1.67 1.00 [user@host bin]$
Output of iostat with sample time 2 seconds: [user@host bin]$ iostat -x /dev/hda /dev/hdb /dev/sda /dev/sdb 2 9999 Linux 2.4.20-8 (host) 03/07/03 avg-cpu: %user %nice %sys %idle 42.02 0.00 4.37 53.61 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 34.78 13.18 53.69 25.53 707.67 310.13 353.84 155.06 12.85 8.17 31.45 12.59 99.70 /dev/hdb 67.52 27.88 10.89 5.15 627.21 264.38 313.61 132.19 55.61 1.45 44.79 62.34 99.95 /dev/sda 143.03 38.59 11.11 1.50 1233.10 320.71 616.55 160.35 123.21 0.76 6.03 3.37 4.25 /dev/sdb 212.68 37.32 13.19 1.23 1806.94 308.38 903.47 154.19 146.73 0.77 5.33 3.05 4.40 avg-cpu: %user %nice %sys %idle 21.50 0.00 1.50 77.00 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 19.50 6.50 29.50 78.50 392.00 680.00 196.00 340.00 9.93 112.05 202.55 9.26 100.00 /dev/hdb 415.00 262.50 61.00 44.00 3800.00 2452.00 1900.00 1226.00 59.54 21474774.83 64.38 9.52 100.00 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 59.50 0.00 5.00 0.00 516.00 0.00 258.00 0.00 103.20 0.10 2.00 2.00 1.00 avg-cpu: %user %nice %sys %idle 8.50 0.00 2.50 89.00 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 109.50 0.00 42.00 0.00 1200.00 0.00 600.00 0.00 28.57 21474735.63 30.36 23.81 100.00 /dev/hdb 1033.50 0.00 157.00 0.00 9524.00 0.00 4762.00 0.00 60.66 21474719.23 8.76 6.37 100.00 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 1108.50 0.00 39.50 0.00 9184.00 0.00 4592.00 0.00 232.51 0.95 2.41 2.41 9.50 avg-cpu: %user %nice %sys %idle 1.00 0.00 2.00 97.00 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 1.00 25.50 7.50 36.00 76.00 612.00 38.00 306.00 15.82 21474790.78 75.75 22.99 100.00 /dev/hdb 1214.50 414.50 189.50 40.00 11232.00 3872.00 5616.00 1936.00 65.81 21474775.03 18.17 4.36 100.00 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 155.00 0.00 8.50 0.00 1308.00 0.00 654.00 0.00 153.88 0.25 2.94 2.94 2.50 [user@host bin]$
[user@host bin]$ iostat -x /dev/hda /dev/hdb /dev/sda /dev/sdb 3 9999 Linux 2.4.20-8 host) 03/07/03 avg-cpu: %user %nice %sys %idle 41.63 0.00 4.35 54.01 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 34.51 13.05 53.39 25.57 703.15 309.46 351.58 154.73 12.82 7.77 32.50 12.63 99.70 /dev/hdb 75.81 31.29 12.26 5.82 704.48 297.01 352.24 148.50 55.39 0.86 44.91 55.28 99.95 /dev/sda 141.27 38.12 10.98 1.48 1217.93 316.76 608.96 158.38 123.21 0.75 6.03 3.37 4.20 /dev/sdb 214.23 36.86 13.19 1.21 1819.31 304.58 909.66 152.29 147.51 0.76 5.30 3.04 4.38 avg-cpu: %user %nice %sys %idle 0.67 0.00 0.67 98.67 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 0.00 0.00 0.33 0.00 2.67 0.00 1.33 0.00 8.00 14316447.69 10.00 3000.00 100.00 /dev/hdb 1036.00 20.67 189.67 6.00 9802.67 218.67 4901.33 109.33 51.22 14316439.75 6.17 5.11 100.00 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 739.67 0.00 26.00 0.00 6125.33 0.00 3062.67 0.00 235.59 0.53 2.05 2.05 5.33 avg-cpu: %user %nice %sys %idle 0.33 0.00 1.33 98.33 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 0.00 2.67 0.00 0.67 0.00 26.67 0.00 13.33 40.00 14316447.72 10.00 1500.00 100.00 /dev/hdb 677.33 635.00 114.00 133.00 6330.67 6144.00 3165.33 3072.00 50.50 78.10 84.24 4.04 99.67 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 78.67 0.00 3.00 0.00 653.33 0.00 326.67 0.00 217.78 0.03 1.11 1.11 0.33 avg-cpu: %user %nice %sys %idle 0.67 0.00 5.00 94.33 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 0.00 0.33 0.00 0.67 0.00 8.00 0.00 4.00 12.00 14316447.69 5.00 1500.00 100.00 /dev/hdb 865.33 572.67 131.00 95.67 7970.67 5402.67 3985.33 2701.33 59.00 14316542.25 45.15 4.41 100.00 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 734.00 0.00 51.00 0.00 6280.00 0.00 3140.00 0.00 123.14 6.10 11.96 2.42 12.33 [user@host bin]$
Note that /dev/hdb was under load at the time these samples were taken, but /dev/hda was not, I believe, under any significant load.
Hmm, new and interesting stats :) Under load, I'm now seeing %util as low as 62.59 and as high as 128.37!!! Now that's impressive.. a disk that is 128% utilised :) Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 386.56 6.32 177.47 11.86 4569.17 145.45 2284.58 72.73 24.90 108.06 120.96 6.10 115.42 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 16976003.74 0.00 0.00 116.60 /dev/sda 15.81 44.66 2.77 7.11 148.62 414.23 74.31 207.11 56.96 0.08 0.80 0.80 0.79 /dev/sdb 0.40 25.30 66.80 100.79 540.71 1008.70 270.36 504.35 9.25 53.44 31.86 3.09 51.78 avg-cpu: %user %nice %sys %idle 9.52 0.00 7.48 82.99 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 157.82 29.25 117.69 6.80 2160.54 288.44 1080.27 144.22 19.67 115.24 150.27 5.03 62.59 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 29217383.24 0.00 0.00 62.59 /dev/sda 0.00 0.68 0.00 2.72 0.00 27.21 0.00 13.61 10.00 0.00 0.00 0.00 0.00 /dev/sdb 0.00 12.24 112.93 8.16 903.40 163.27 451.70 81.63 8.81 6.19 5.11 4.27 51.70 avg-cpu: %user %nice %sys %idle 47.52 0.00 6.44 46.04 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 428.71 10.40 158.42 24.26 4685.15 277.23 2342.57 138.61 27.17 120.84 135.80 6.12 111.88 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 21262067.60 0.00 0.00 112.87 /dev/sda 6.93 14.36 1.98 1.49 71.29 126.73 35.64 63.37 57.14 0.00 0.00 0.00 0.00 /dev/sdb 0.00 2.48 98.02 62.38 780.20 534.65 390.10 267.33 8.20 6.88 4.32 3.61 57.92 avg-cpu: %user %nice %sys %idle 5.79 0.00 5.02 89.19 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 324.71 16.22 168.34 8.88 3987.64 206.95 1993.82 103.47 23.67 164.05 148.63 5.47 96.91 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 16582759.33 0.00 0.00 96.91 /dev/sda 18.53 25.87 3.09 4.63 172.97 244.02 86.49 122.01 54.00 0.08 1.00 1.00 0.77 /dev/sdb 0.00 27.41 100.00 132.82 800.00 1269.50 400.00 634.75 8.89 106.83 45.89 3.40 79.15 avg-cpu: %user %nice %sys %idle 14.18 0.00 7.80 78.01 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 342.55 1.42 143.26 17.73 3795.74 147.52 1897.87 73.76 24.49 150.43 167.09 5.95 95.74 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 30460636.50 0.00 0.00 95.74 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 0.00 2.84 131.91 30.50 1060.99 266.67 530.50 133.33 8.17 6.95 4.28 3.84 62.41 avg-cpu: %user %nice %sys %idle 13.54 0.00 6.11 80.35 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 440.17 0.00 205.68 1.75 5156.33 10.48 2578.17 5.24 24.91 109.61 110.44 5.05 104.80 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 18755177.76 0.00 0.00 106.11 /dev/sda 7.42 0.44 3.06 1.31 83.84 13.97 41.92 6.99 22.40 0.04 1.00 1.00 0.44 /dev/sdb 0.00 9.17 121.83 105.68 974.67 918.78 487.34 459.39 8.32 10.92 4.78 3.90 88.65 avg-cpu: %user %nice %sys %idle 5.11 0.00 5.68 89.20 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 387.50 6.25 127.27 10.23 4127.27 131.82 2063.64 65.91 30.98 17.90 84.34 6.40 88.07 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 24403106.57 0.00 0.00 89.77 /dev/sda 0.00 0.57 1.14 3.41 9.09 31.82 4.55 15.91 9.00 0.06 1.25 1.25 0.57 /dev/sdb 0.00 39.20 65.91 131.82 522.73 1368.18 261.36 684.09 9.56 184.09 93.13 3.22 63.64 avg-cpu: %user %nice %sys %idle 13.89 0.00 6.94 79.17 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 435.19 0.00 162.96 0.46 4774.07 3.70 2387.04 1.85 29.24 19884081.23 53.12 5.47 89.35 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19883975.44 0.00 0.00 101.85 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 0.00 2.31 87.50 38.89 700.00 329.63 350.00 164.81 8.15 5.09 4.03 3.66 46.30 avg-cpu: %user %nice %sys %idle 6.74 0.00 4.15 89.12 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 399.48 5.18 168.39 2.07 4559.59 58.03 2279.79 29.02 27.09 62.85 106.81 5.99 102.07 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 22253573.76 0.00 0.00 109.84 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 0.52 40.41 101.55 167.36 820.73 1662.18 410.36 831.09 9.23 250.88 93.29 3.03 81.35 avg-cpu: %user %nice %sys %idle 20.44 0.00 7.11 72.44 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 317.33 14.67 165.33 18.22 3879.11 266.67 1939.56 133.33 22.59 92.31 115.86 5.93 108.89 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19088597.36 0.00 0.00 112.44 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 0.00 2.67 164.00 49.78 1312.00 419.56 656.00 209.78 8.10 9.82 4.57 3.91 83.56 avg-cpu: %user %nice %sys %idle 45.03 0.00 7.85 47.12 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 251.31 10.47 117.80 8.90 2923.56 150.79 1461.78 75.39 24.26 45.60 103.02 5.58 70.68 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 22486645.16 0.00 0.00 72.77 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 0.00 11.52 92.67 59.69 741.36 569.63 370.68 284.82 8.60 6.60 4.36 3.75 57.07 avg-cpu: %user %nice %sys %idle 14.90 0.00 5.77 79.33 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 374.04 21.63 212.50 16.83 4730.77 323.08 2365.38 161.54 22.04 130.96 112.43 5.35 122.60 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 20648714.36 0.00 0.00 128.37 /dev/sda 0.00 23.08 3.37 2.40 26.92 203.85 13.46 101.92 40.00 0.00 0.00 0.00 0.00 /dev/sdb 0.00 44.71 148.56 175.00 1188.46 1769.23 594.23 884.62 9.14 173.51 53.61 3.15 101.92 avg-cpu: %user %nice %sys %idle 21.50 0.00 8.00 70.50 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 181.50 0.00 118.00 7.00 2380.00 44.00 1190.00 22.00 19.39 92.05 146.20 5.88 73.50 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 21474740.28 0.00 0.00 74.00 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 0.00 1.50 85.00 30.00 676.00 240.00 338.00 120.00 7.97 5.55 4.87 3.91 45.00 avg-cpu: %user %nice %sys %idle 18.08 0.00 4.80 77.12 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 313.65 14.39 169.74 13.28 3837.64 218.45 1918.82 109.23 22.16 64.76 102.30 5.56 101.85 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 15848449.17 0.00 0.00 106.64 /dev/sda 0.00 2.21 1.48 2.58 11.81 38.38 5.90 19.19 12.36 0.00 0.00 0.00 0.00 /dev/sdb 0.00 51.66 112.55 171.22 903.32 1783.03 451.66 891.51 9.47 260.30 91.72 3.20 90.77 avg-cpu: %user %nice %sys %idle 15.76 0.00 5.45 78.79 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 145.45 0.00 101.82 13.33 2050.91 106.67 1025.45 53.33 18.74 26030095.07 66.79 6.58 75.76 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 26030000.04 0.00 0.00 80.61 /dev/sda 0.00 1.21 1.21 0.61 9.70 14.55 4.85 7.27 13.33 0.00 0.00 0.00 0.00 /dev/sdb 0.00 1.21 136.97 20.00 1095.76 169.70 547.88 84.85 8.06 7.94 5.06 4.21 66.06 avg-cpu: %user %nice %sys %idle 9.30 0.00 5.58 85.12 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 440.47 0.00 195.35 3.72 5026.98 29.77 2513.49 14.88 25.40 19976578.77 54.95 5.19 103.26 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19976449.38 0.00 0.00 109.77 /dev/sda 0.00 14.42 0.00 2.33 0.00 133.95 0.00 66.98 57.60 0.00 0.00 0.00 0.00 /dev/sdb 0.00 15.35 133.02 120.47 1064.19 1086.51 532.09 543.26 8.48 14.93 5.82 3.58 90.70 avg-cpu: %user %nice %sys %idle 14.14 0.00 5.24 80.63 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 241.88 10.47 112.57 9.95 2848.17 167.54 1424.08 83.77 24.62 22486705.84 50.34 6.50 79.58 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 22486624.06 0.00 0.00 89.01 /dev/sda 0.00 3.14 1.05 3.14 8.38 50.26 4.19 25.13 14.00 0.00 0.00 0.00 0.00 /dev/sdb 0.00 42.93 105.76 136.13 846.07 1432.46 423.04 716.23 9.42 241.73 100.02 3.25 78.53 avg-cpu: %user %nice %sys %idle 25.85 0.00 7.32 66.83 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 218.54 3.41 173.17 4.88 3118.05 62.44 1559.02 31.22 17.86 5.95 69.10 5.67 100.98 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 20950924.27 0.00 0.00 104.39 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 0.00 4.39 144.39 46.34 1155.12 405.85 577.56 202.93 8.18 9.37 4.91 3.99 76.10 avg-cpu: %user %nice %sys %idle 10.50 0.00 4.00 85.50 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 226.00 28.00 117.00 17.50 2768.00 372.00 1384.00 186.00 23.35 21474817.48 61.23 6.91 93.00 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 21474712.33 0.00 0.00 95.50 /dev/sda 12.50 2.50 3.00 3.00 124.00 44.00 62.00 22.00 28.00 0.00 0.00 0.00 0.00 /dev/sdb 0.00 44.50 65.00 217.00 520.00 2124.00 260.00 1062.00 9.38 320.45 105.25 3.05 86.00
I have seen this behavior on our RHEL 3.0 system as well. It's very concerning to me, mainly because I think on some level this is effecting the hard drive performance. Judging solely by the "little yellow" hard drive activity light (that once flickered when it was active) it is now solid amber, with no inactivity. My concern is that the problem is continually spinning the disk drive(s). As previously stated, I can reproduce every time, as well as what Michael Wright noticed, the avgqu-sz does seem to be related to the 'duration' of iostat. I feel that the problem is deeper than just a problem with iostat.
I can reproduce this behaviour on a number of Dell Poweredge machines (HyperThreading) running RH versions 7.1, 7.2, 7.3, 8.0 and 9.0 with stock kernels. I see the exact same results as the other commenters.
*** Bug 117930 has been marked as a duplicate of this bug. ***
As RHL9 is dead, and the same thing happens on RHEL3, reassigning.
The version of sysstat affected is 4.0.7-4.EL3.2
The problem only happens with IDE disks, not with SCSI devices.
That (SCSI disks are not affected) was my experience also. Note too that not all systems or all IDE disks are affected as well, I have a RedHat 7.2 system with a single IDE disk which does not exhibit the problem.
Created attachment 103097 [details] patch to correct aveq/util computation in iostat i'm not 100% sure of this, but I think that /proc/partitions (which is used in the beta for iostat) is miscomputing aveq-sz and utilization. The kernel tracks these values in proc/partitions as differences between idle times, and so I don't think an extra difference computation in iostat is needed. Also the utilization values in /proc/partitions are given in ticks, so I believe that util computation was off by a factor of 10. This patch (applied to the U3 beta of sysstat) fixes those problems.
The from Comment #13 appears to cause %util numbers that are too low. Here is iostat -x 1 on an unloaded disk. The avgqu-sz is too high. The svctm is too high (intermitantly), and the %util seems low. avg-cpu: %user %nice %sys %idle 4.95 0.00 0.99 94.06 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 0.00 8.91 0.00 3.96 0.00 102.97 0.00 51.49 26.00 348700.83 0.00 252.50 0.10 Here is the same iostat -x 1 command with a loaded disk (cat /dev/hda > /dev/null): avg-cpu: %user %nice %sys %idle 100.00 0.00 0.00 0.00 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 18614.29 814.29 1114.29 71.43 48228.57 7085.71 24114.29 3542.86 46.65 3772660.57 90.84 12.05 1.43 Again, the %util is low as the disk is heavily loaded, the avgqu-sz is really up there, and the svctm seems pretty reasonable.
*** This bug has been marked as a duplicate of 134579 ***
Fixing component and assignee.
Propagating acks from bug 134579. *** This bug has been marked as a duplicate of 134579 ***
An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on the solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2005-663.html