Bug 98542
Summary: | iostat -x shows infeasible avgqu-sz results and max util | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 3 | Reporter: | Michael Wright <michael> | ||||
Component: | kernel | Assignee: | Larry Woodman <lwoodman> | ||||
Status: | CLOSED ERRATA | QA Contact: | Brian Brock <bbrock> | ||||
Severity: | medium | Docs Contact: | |||||
Priority: | medium | ||||||
Version: | 3.0 | CC: | fabud, k.georgiou, lwang, michael, nphilipp, peterm, petrides, tao | ||||
Target Milestone: | --- | ||||||
Target Release: | --- | ||||||
Hardware: | i686 | ||||||
OS: | Linux | ||||||
Whiteboard: | |||||||
Fixed In Version: | RHSA-2005-663 | Doc Type: | Bug Fix | ||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | Environment: | ||||||
Last Closed: | 2005-09-28 14:18:12 UTC | Type: | --- | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Bug Depends On: | |||||||
Bug Blocks: | 156321 | ||||||
Attachments: |
|
Description
Michael Wright
2003-07-03 13:51:54 UTC
Output of iostat with sample time 1 second: [user@host bin]$ iostat -x /dev/hda /dev/hdb /dev/sda /dev/sdb 1 9999 Linux 2.4.20-8 (radiated.local.bogusmove.com) 03/07/03 avg-cpu: %user %nice %sys %idle 42.46 0.00 4.38 53.15 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 35.03 13.32 54.12 25.58 713.16 311.67 356.58 155.83 12.86 8.80 30.64 12.51 99.70 /dev/hdb 58.71 24.13 9.49 4.44 545.50 228.70 272.75 114.35 55.57 1.90 44.01 71.74 99.95 /dev/sda 144.75 39.06 11.25 1.52 1247.96 324.57 623.98 162.29 123.21 0.77 6.03 3.37 4.31 /dev/sdb 211.29 37.77 13.17 1.24 1795.65 312.09 897.83 156.05 146.27 0.77 5.37 3.06 4.42 avg-cpu: %user %nice %sys %idle 1.00 0.00 3.00 96.00 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 0.00 8.00 0.00 2.00 0.00 80.00 0.00 40.00 40.00 42949563.06 5.00 500.00 100.00 /dev/hdb 692.00 1214.00 113.00 204.00 6440.00 11616.00 3220.00 5808.00 56.96 79.00 46.72 3.15 100.00 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 68.00 0.00 5.00 0.00 584.00 0.00 292.00 0.00 116.80 0.10 2.00 2.00 1.00 avg-cpu: %user %nice %sys %idle 0.00 0.00 4.00 96.00 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 42949562.96 0.00 0.00 100.00 /dev/hdb 588.00 809.00 99.00 155.00 5496.00 7440.00 2748.00 3720.00 50.93 94.50 112.40 3.90 99.00 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 66.00 0.00 10.00 0.00 608.00 0.00 304.00 0.00 60.80 0.30 3.00 3.00 3.00 avg-cpu: %user %nice %sys %idle 0.00 0.00 0.00 100.00 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 42949562.96 0.00 0.00 100.00 /dev/hdb 1102.00 0.00 187.00 0.00 10312.00 0.00 5156.00 0.00 55.14 42949555.26 6.63 5.35 100.00 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 162.00 0.00 16.00 0.00 1424.00 0.00 712.00 0.00 89.00 0.60 3.75 2.50 4.00 avg-cpu: %user %nice %sys %idle 0.00 0.00 0.00 100.00 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 42949562.96 0.00 0.00 100.00 /dev/hdb 1150.00 0.00 199.00 0.00 10792.00 0.00 5396.00 0.00 54.23 42949554.76 5.93 5.03 100.00 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 128.00 0.00 6.00 0.00 1072.00 0.00 536.00 0.00 178.67 0.10 1.67 1.67 1.00 [user@host bin]$ Output of iostat with sample time 2 seconds: [user@host bin]$ iostat -x /dev/hda /dev/hdb /dev/sda /dev/sdb 2 9999 Linux 2.4.20-8 (host) 03/07/03 avg-cpu: %user %nice %sys %idle 42.02 0.00 4.37 53.61 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 34.78 13.18 53.69 25.53 707.67 310.13 353.84 155.06 12.85 8.17 31.45 12.59 99.70 /dev/hdb 67.52 27.88 10.89 5.15 627.21 264.38 313.61 132.19 55.61 1.45 44.79 62.34 99.95 /dev/sda 143.03 38.59 11.11 1.50 1233.10 320.71 616.55 160.35 123.21 0.76 6.03 3.37 4.25 /dev/sdb 212.68 37.32 13.19 1.23 1806.94 308.38 903.47 154.19 146.73 0.77 5.33 3.05 4.40 avg-cpu: %user %nice %sys %idle 21.50 0.00 1.50 77.00 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 19.50 6.50 29.50 78.50 392.00 680.00 196.00 340.00 9.93 112.05 202.55 9.26 100.00 /dev/hdb 415.00 262.50 61.00 44.00 3800.00 2452.00 1900.00 1226.00 59.54 21474774.83 64.38 9.52 100.00 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 59.50 0.00 5.00 0.00 516.00 0.00 258.00 0.00 103.20 0.10 2.00 2.00 1.00 avg-cpu: %user %nice %sys %idle 8.50 0.00 2.50 89.00 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 109.50 0.00 42.00 0.00 1200.00 0.00 600.00 0.00 28.57 21474735.63 30.36 23.81 100.00 /dev/hdb 1033.50 0.00 157.00 0.00 9524.00 0.00 4762.00 0.00 60.66 21474719.23 8.76 6.37 100.00 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 1108.50 0.00 39.50 0.00 9184.00 0.00 4592.00 0.00 232.51 0.95 2.41 2.41 9.50 avg-cpu: %user %nice %sys %idle 1.00 0.00 2.00 97.00 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 1.00 25.50 7.50 36.00 76.00 612.00 38.00 306.00 15.82 21474790.78 75.75 22.99 100.00 /dev/hdb 1214.50 414.50 189.50 40.00 11232.00 3872.00 5616.00 1936.00 65.81 21474775.03 18.17 4.36 100.00 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 155.00 0.00 8.50 0.00 1308.00 0.00 654.00 0.00 153.88 0.25 2.94 2.94 2.50 [user@host bin]$ [user@host bin]$ iostat -x /dev/hda /dev/hdb /dev/sda /dev/sdb 3 9999 Linux 2.4.20-8 host) 03/07/03 avg-cpu: %user %nice %sys %idle 41.63 0.00 4.35 54.01 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 34.51 13.05 53.39 25.57 703.15 309.46 351.58 154.73 12.82 7.77 32.50 12.63 99.70 /dev/hdb 75.81 31.29 12.26 5.82 704.48 297.01 352.24 148.50 55.39 0.86 44.91 55.28 99.95 /dev/sda 141.27 38.12 10.98 1.48 1217.93 316.76 608.96 158.38 123.21 0.75 6.03 3.37 4.20 /dev/sdb 214.23 36.86 13.19 1.21 1819.31 304.58 909.66 152.29 147.51 0.76 5.30 3.04 4.38 avg-cpu: %user %nice %sys %idle 0.67 0.00 0.67 98.67 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 0.00 0.00 0.33 0.00 2.67 0.00 1.33 0.00 8.00 14316447.69 10.00 3000.00 100.00 /dev/hdb 1036.00 20.67 189.67 6.00 9802.67 218.67 4901.33 109.33 51.22 14316439.75 6.17 5.11 100.00 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 739.67 0.00 26.00 0.00 6125.33 0.00 3062.67 0.00 235.59 0.53 2.05 2.05 5.33 avg-cpu: %user %nice %sys %idle 0.33 0.00 1.33 98.33 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 0.00 2.67 0.00 0.67 0.00 26.67 0.00 13.33 40.00 14316447.72 10.00 1500.00 100.00 /dev/hdb 677.33 635.00 114.00 133.00 6330.67 6144.00 3165.33 3072.00 50.50 78.10 84.24 4.04 99.67 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 78.67 0.00 3.00 0.00 653.33 0.00 326.67 0.00 217.78 0.03 1.11 1.11 0.33 avg-cpu: %user %nice %sys %idle 0.67 0.00 5.00 94.33 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 0.00 0.33 0.00 0.67 0.00 8.00 0.00 4.00 12.00 14316447.69 5.00 1500.00 100.00 /dev/hdb 865.33 572.67 131.00 95.67 7970.67 5402.67 3985.33 2701.33 59.00 14316542.25 45.15 4.41 100.00 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 734.00 0.00 51.00 0.00 6280.00 0.00 3140.00 0.00 123.14 6.10 11.96 2.42 12.33 [user@host bin]$ Note that /dev/hdb was under load at the time these samples were taken, but /dev/hda was not, I believe, under any significant load. Hmm, new and interesting stats :) Under load, I'm now seeing %util as low as 62.59 and as high as 128.37!!! Now that's impressive.. a disk that is 128% utilised :) Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 386.56 6.32 177.47 11.86 4569.17 145.45 2284.58 72.73 24.90 108.06 120.96 6.10 115.42 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 16976003.74 0.00 0.00 116.60 /dev/sda 15.81 44.66 2.77 7.11 148.62 414.23 74.31 207.11 56.96 0.08 0.80 0.80 0.79 /dev/sdb 0.40 25.30 66.80 100.79 540.71 1008.70 270.36 504.35 9.25 53.44 31.86 3.09 51.78 avg-cpu: %user %nice %sys %idle 9.52 0.00 7.48 82.99 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 157.82 29.25 117.69 6.80 2160.54 288.44 1080.27 144.22 19.67 115.24 150.27 5.03 62.59 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 29217383.24 0.00 0.00 62.59 /dev/sda 0.00 0.68 0.00 2.72 0.00 27.21 0.00 13.61 10.00 0.00 0.00 0.00 0.00 /dev/sdb 0.00 12.24 112.93 8.16 903.40 163.27 451.70 81.63 8.81 6.19 5.11 4.27 51.70 avg-cpu: %user %nice %sys %idle 47.52 0.00 6.44 46.04 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 428.71 10.40 158.42 24.26 4685.15 277.23 2342.57 138.61 27.17 120.84 135.80 6.12 111.88 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 21262067.60 0.00 0.00 112.87 /dev/sda 6.93 14.36 1.98 1.49 71.29 126.73 35.64 63.37 57.14 0.00 0.00 0.00 0.00 /dev/sdb 0.00 2.48 98.02 62.38 780.20 534.65 390.10 267.33 8.20 6.88 4.32 3.61 57.92 avg-cpu: %user %nice %sys %idle 5.79 0.00 5.02 89.19 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 324.71 16.22 168.34 8.88 3987.64 206.95 1993.82 103.47 23.67 164.05 148.63 5.47 96.91 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 16582759.33 0.00 0.00 96.91 /dev/sda 18.53 25.87 3.09 4.63 172.97 244.02 86.49 122.01 54.00 0.08 1.00 1.00 0.77 /dev/sdb 0.00 27.41 100.00 132.82 800.00 1269.50 400.00 634.75 8.89 106.83 45.89 3.40 79.15 avg-cpu: %user %nice %sys %idle 14.18 0.00 7.80 78.01 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 342.55 1.42 143.26 17.73 3795.74 147.52 1897.87 73.76 24.49 150.43 167.09 5.95 95.74 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 30460636.50 0.00 0.00 95.74 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 0.00 2.84 131.91 30.50 1060.99 266.67 530.50 133.33 8.17 6.95 4.28 3.84 62.41 avg-cpu: %user %nice %sys %idle 13.54 0.00 6.11 80.35 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 440.17 0.00 205.68 1.75 5156.33 10.48 2578.17 5.24 24.91 109.61 110.44 5.05 104.80 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 18755177.76 0.00 0.00 106.11 /dev/sda 7.42 0.44 3.06 1.31 83.84 13.97 41.92 6.99 22.40 0.04 1.00 1.00 0.44 /dev/sdb 0.00 9.17 121.83 105.68 974.67 918.78 487.34 459.39 8.32 10.92 4.78 3.90 88.65 avg-cpu: %user %nice %sys %idle 5.11 0.00 5.68 89.20 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 387.50 6.25 127.27 10.23 4127.27 131.82 2063.64 65.91 30.98 17.90 84.34 6.40 88.07 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 24403106.57 0.00 0.00 89.77 /dev/sda 0.00 0.57 1.14 3.41 9.09 31.82 4.55 15.91 9.00 0.06 1.25 1.25 0.57 /dev/sdb 0.00 39.20 65.91 131.82 522.73 1368.18 261.36 684.09 9.56 184.09 93.13 3.22 63.64 avg-cpu: %user %nice %sys %idle 13.89 0.00 6.94 79.17 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 435.19 0.00 162.96 0.46 4774.07 3.70 2387.04 1.85 29.24 19884081.23 53.12 5.47 89.35 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19883975.44 0.00 0.00 101.85 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 0.00 2.31 87.50 38.89 700.00 329.63 350.00 164.81 8.15 5.09 4.03 3.66 46.30 avg-cpu: %user %nice %sys %idle 6.74 0.00 4.15 89.12 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 399.48 5.18 168.39 2.07 4559.59 58.03 2279.79 29.02 27.09 62.85 106.81 5.99 102.07 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 22253573.76 0.00 0.00 109.84 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 0.52 40.41 101.55 167.36 820.73 1662.18 410.36 831.09 9.23 250.88 93.29 3.03 81.35 avg-cpu: %user %nice %sys %idle 20.44 0.00 7.11 72.44 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 317.33 14.67 165.33 18.22 3879.11 266.67 1939.56 133.33 22.59 92.31 115.86 5.93 108.89 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19088597.36 0.00 0.00 112.44 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 0.00 2.67 164.00 49.78 1312.00 419.56 656.00 209.78 8.10 9.82 4.57 3.91 83.56 avg-cpu: %user %nice %sys %idle 45.03 0.00 7.85 47.12 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 251.31 10.47 117.80 8.90 2923.56 150.79 1461.78 75.39 24.26 45.60 103.02 5.58 70.68 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 22486645.16 0.00 0.00 72.77 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 0.00 11.52 92.67 59.69 741.36 569.63 370.68 284.82 8.60 6.60 4.36 3.75 57.07 avg-cpu: %user %nice %sys %idle 14.90 0.00 5.77 79.33 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 374.04 21.63 212.50 16.83 4730.77 323.08 2365.38 161.54 22.04 130.96 112.43 5.35 122.60 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 20648714.36 0.00 0.00 128.37 /dev/sda 0.00 23.08 3.37 2.40 26.92 203.85 13.46 101.92 40.00 0.00 0.00 0.00 0.00 /dev/sdb 0.00 44.71 148.56 175.00 1188.46 1769.23 594.23 884.62 9.14 173.51 53.61 3.15 101.92 avg-cpu: %user %nice %sys %idle 21.50 0.00 8.00 70.50 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 181.50 0.00 118.00 7.00 2380.00 44.00 1190.00 22.00 19.39 92.05 146.20 5.88 73.50 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 21474740.28 0.00 0.00 74.00 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 0.00 1.50 85.00 30.00 676.00 240.00 338.00 120.00 7.97 5.55 4.87 3.91 45.00 avg-cpu: %user %nice %sys %idle 18.08 0.00 4.80 77.12 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 313.65 14.39 169.74 13.28 3837.64 218.45 1918.82 109.23 22.16 64.76 102.30 5.56 101.85 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 15848449.17 0.00 0.00 106.64 /dev/sda 0.00 2.21 1.48 2.58 11.81 38.38 5.90 19.19 12.36 0.00 0.00 0.00 0.00 /dev/sdb 0.00 51.66 112.55 171.22 903.32 1783.03 451.66 891.51 9.47 260.30 91.72 3.20 90.77 avg-cpu: %user %nice %sys %idle 15.76 0.00 5.45 78.79 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 145.45 0.00 101.82 13.33 2050.91 106.67 1025.45 53.33 18.74 26030095.07 66.79 6.58 75.76 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 26030000.04 0.00 0.00 80.61 /dev/sda 0.00 1.21 1.21 0.61 9.70 14.55 4.85 7.27 13.33 0.00 0.00 0.00 0.00 /dev/sdb 0.00 1.21 136.97 20.00 1095.76 169.70 547.88 84.85 8.06 7.94 5.06 4.21 66.06 avg-cpu: %user %nice %sys %idle 9.30 0.00 5.58 85.12 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 440.47 0.00 195.35 3.72 5026.98 29.77 2513.49 14.88 25.40 19976578.77 54.95 5.19 103.26 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 19976449.38 0.00 0.00 109.77 /dev/sda 0.00 14.42 0.00 2.33 0.00 133.95 0.00 66.98 57.60 0.00 0.00 0.00 0.00 /dev/sdb 0.00 15.35 133.02 120.47 1064.19 1086.51 532.09 543.26 8.48 14.93 5.82 3.58 90.70 avg-cpu: %user %nice %sys %idle 14.14 0.00 5.24 80.63 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 241.88 10.47 112.57 9.95 2848.17 167.54 1424.08 83.77 24.62 22486705.84 50.34 6.50 79.58 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 22486624.06 0.00 0.00 89.01 /dev/sda 0.00 3.14 1.05 3.14 8.38 50.26 4.19 25.13 14.00 0.00 0.00 0.00 0.00 /dev/sdb 0.00 42.93 105.76 136.13 846.07 1432.46 423.04 716.23 9.42 241.73 100.02 3.25 78.53 avg-cpu: %user %nice %sys %idle 25.85 0.00 7.32 66.83 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 218.54 3.41 173.17 4.88 3118.05 62.44 1559.02 31.22 17.86 5.95 69.10 5.67 100.98 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 20950924.27 0.00 0.00 104.39 /dev/sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 /dev/sdb 0.00 4.39 144.39 46.34 1155.12 405.85 577.56 202.93 8.18 9.37 4.91 3.99 76.10 avg-cpu: %user %nice %sys %idle 10.50 0.00 4.00 85.50 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 226.00 28.00 117.00 17.50 2768.00 372.00 1384.00 186.00 23.35 21474817.48 61.23 6.91 93.00 /dev/hdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 21474712.33 0.00 0.00 95.50 /dev/sda 12.50 2.50 3.00 3.00 124.00 44.00 62.00 22.00 28.00 0.00 0.00 0.00 0.00 /dev/sdb 0.00 44.50 65.00 217.00 520.00 2124.00 260.00 1062.00 9.38 320.45 105.25 3.05 86.00 I have seen this behavior on our RHEL 3.0 system as well. It's very concerning to me, mainly because I think on some level this is effecting the hard drive performance. Judging solely by the "little yellow" hard drive activity light (that once flickered when it was active) it is now solid amber, with no inactivity. My concern is that the problem is continually spinning the disk drive(s). As previously stated, I can reproduce every time, as well as what Michael Wright noticed, the avgqu-sz does seem to be related to the 'duration' of iostat. I feel that the problem is deeper than just a problem with iostat. I can reproduce this behaviour on a number of Dell Poweredge machines (HyperThreading) running RH versions 7.1, 7.2, 7.3, 8.0 and 9.0 with stock kernels. I see the exact same results as the other commenters. *** Bug 117930 has been marked as a duplicate of this bug. *** As RHL9 is dead, and the same thing happens on RHEL3, reassigning. The version of sysstat affected is 4.0.7-4.EL3.2 The problem only happens with IDE disks, not with SCSI devices. That (SCSI disks are not affected) was my experience also. Note too that not all systems or all IDE disks are affected as well, I have a RedHat 7.2 system with a single IDE disk which does not exhibit the problem. Created attachment 103097 [details]
patch to correct aveq/util computation in iostat
i'm not 100% sure of this, but I think that /proc/partitions (which is used in
the beta for iostat) is miscomputing aveq-sz and utilization. The kernel
tracks these values in proc/partitions as differences between idle times, and
so I don't think an extra difference computation in iostat is needed. Also the
utilization values in /proc/partitions are given in ticks, so I believe that
util computation was off by a factor of 10. This patch (applied to the U3 beta
of sysstat) fixes those problems.
The from Comment #13 appears to cause %util numbers that are too low. Here is iostat -x 1 on an unloaded disk. The avgqu-sz is too high. The svctm is too high (intermitantly), and the %util seems low. avg-cpu: %user %nice %sys %idle 4.95 0.00 0.99 94.06 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 0.00 8.91 0.00 3.96 0.00 102.97 0.00 51.49 26.00 348700.83 0.00 252.50 0.10 Here is the same iostat -x 1 command with a loaded disk (cat /dev/hda > /dev/null): avg-cpu: %user %nice %sys %idle 100.00 0.00 0.00 0.00 Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util /dev/hda 18614.29 814.29 1114.29 71.43 48228.57 7085.71 24114.29 3542.86 46.65 3772660.57 90.84 12.05 1.43 Again, the %util is low as the disk is heavily loaded, the avgqu-sz is really up there, and the svctm seems pretty reasonable. *** This bug has been marked as a duplicate of 134579 *** Fixing component and assignee. Propagating acks from bug 134579. *** This bug has been marked as a duplicate of 134579 *** An advisory has been issued which should help the problem described in this bug report. This report is therefore being closed with a resolution of ERRATA. For more information on the solution and/or where to find the updated files, please follow the link below. You may reopen this bug report if the solution does not work for you. http://rhn.redhat.com/errata/RHSA-2005-663.html |