Bug 1859048

Summary: qemu-kvm-5.0.0-0.module+el8.3.0+7323 introduces block performance regression for localfs backend
Product: Red Hat Enterprise Linux 9 Reporter: Yanhui Ma <yama>
Component: qemu-kvmAssignee: Stefan Hajnoczi <stefanha>
qemu-kvm sub component: virtio-blk,scsi QA Contact: Tingting Mao <timao>
Status: CLOSED WONTFIX Docs Contact:
Severity: high    
Priority: high CC: aliang, alitke, chayang, coli, fdeutsch, jinzhao, juzhang, kwolf, mprivozn, mrashish, sgarzare, smitterl, stefanha, timao, vgoyal, virt-maint, zhenyzha
Version: 9.1Keywords: Regression, Reopened, Triaged
Target Milestone: rc   
Target Release: 9.3   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-04-26 12:41:39 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1897025, 1948358    

Description Yanhui Ma 2020-07-21 08:18:51 UTC 
Description of problem:

When comparing rhel8.3.0 block performance with rhel8.2.1, there are ~40% performance regression for 4k 64iodepth and 16k 64iodepth read and write, only for raw+blk and qcow2+blk. Here are detailed results:

http://kvm-perf.englab.nay.redhat.com/results/regression/block8.3.0_qemu5.0.0-0.scrmod-7066_kernel217round1/linux/xfs/qcow2.virtio_blk.*.x86_64.html-----regression
http://kvm-perf.englab.nay.redhat.com/results/regression/block8.3.0_qemu5.0.0-0.scrmod-7066_kernel217round1/linux/xfs/raw.virtio_blk.*.x86_64.html----regression
http://kvm-perf.englab.nay.redhat.com/results/regression/block8.3.0_qemu5.0.0-0.scrmod-7066_kernel217round1/linux/xfs/qcow2.virtio_scsi.*.x86_64.html------no regression
http://kvm-perf.englab.nay.redhat.com/results/regression/block8.3.0_qemu5.0.0-0.scrmod-7066_kernel217round1/linux/xfs/raw.virtio_scsi.*.x86_64.html-------no regression

After analyzing, we found qemu-kvm-5.0.0-0.module+el8.3.0+7323 causes the regression, here are comparision results between different qemu:
http://kvm-perf.englab.nay.redhat.com/results/regression/block8.3.0_qemu5.0.0-0.scrmod-7066_kernel217round1/linux/xfs/diffqemu/qcow2.virtio_blk.*.x86_64.html

Version-Release number of selected component (if applicable):
qemu-kvm-5.0.0-0.module+el8.3.0+7323+d54bb644.wrb200701.x86_64'
host kernel:  4.18.0-217.el8.x86_64
guest kernel: 4.18.0-217.el8.x86_64

How reproducible:
100%

Steps to Reproduce:
1. on host:
#mkfs.xfs /dev/sdb & mount /dev/sdb /mnt/test
/dev/sdb is a 60GB ssd.
#qemu-img create -f qcow2 /mnt/test/storage2.qcow2 40G

2. boot a guest with above image:
MALLOC_PERTURB_=1 numactl \
    -m 1  /usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -machine q35 \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 4096  \
    -smp 2,maxcpus=2,cores=1,threads=1,dies=1,sockets=2  \
    -cpu 'IvyBridge',+kvm_pv_unhalt \
    -chardev socket,server,id=qmp_id_qmpmonitor1,nowait,path=/var/tmp/avocado_bapfdqao/monitor-qmpmonitor1-20200721-014154-5HJGMjxW  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,server,id=qmp_id_catch_monitor,nowait,path=/var/tmp/avocado_bapfdqao/monitor-catch_monitor-20200721-014154-5HJGMjxW  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=id31BN83 \
    -chardev socket,server,id=chardev_serial0,nowait,path=/var/tmp/avocado_bapfdqao/serial-serial0-20200721-014154-5HJGMjxW \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20200721-014154-5HJGMjxW,path=/var/tmp/avocado_bapfdqao/seabios-20200721-014154-5HJGMjxW,server,nowait \
    -device isa-debugcon,chardev=seabioslog_id_20200721-014154-5HJGMjxW,iobase=0x402 \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/rhel830-64-virtio.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,write-cache=on,bus=pcie-root-port-2,addr=0x0 \
    -blockdev node-name=file_disk1,driver=file,aio=threads,filename=/mnt/test/storage2.qcow2,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_disk1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_disk1 \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-blk-pci,id=disk1,drive=drive_disk1,bootindex=1,write-cache=on,bus=pcie-root-port-3,addr=0x0 \
    -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x1.0x4,bus=pcie.0,chassis=5 \
    -device virtio-net-pci,mac=9a:37:37:37:37:4e,id=idBMd7vy,netdev=idLb51aS,bus=pcie-root-port-4,addr=0x0  \
    -netdev tap,id=idLb51aS,fd=14  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=6

3. on host:
# numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23
node 0 size: 31772 MB
node 0 free: 31130 MB
node 1 cpus: 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31
node 1 size: 32227 MB
node 1 free: 30543 MB
node distances:
node   0   1 
  0:  10  11 
  1:  11  10 

Pin two vcpu threads to cpu(8) and cpu(9) respectively.

4. running fio on the data disk created by step1 on guest:
#mkfs.xfs /dev/vdb
#mount /dev/vdb /mnt
#fio --rw=%s --bs=%s --iodepth=%s --runtime=1m --direct=1 --filename=/mnt/%s --name=job1 --ioengine=libaio --thread --group_reporting --numjobs=16 --size=512MB --time_based --output=/tmp/fio_result &> /dev/null

- rw: read write randread randwrite randrw
- bs: 4k 16k 64k 256k
- iodepth: 1 8 64

Actual results:

There are ~40% performance regression for 4k 64iodepth and 16k 64iodepth read and write, only for raw+blk and qcow2+blk.  

Expected results:

No regression
Additional info:
Comment 2 Stefan Hajnoczi 2020-07-28 13:24:49 UTC 
I have not been able to reproduce a significant regression yet but will keep trying.

Does -machine pc-q35-rhel8.2.0 change the results with qemu-kvm-5.0.0-0.module+el8.3.0+7323+d54bb644.wrb200701.x86_64?

The pc-q35-rhel8.3.0 machine type includes changes to the virtio-blk queue size. This can affect the benchmark since it uses a high queue size.
Comment 4 Yanhui Ma 2020-08-05 02:14:26 UTC 
(In reply to Stefan Hajnoczi from comment #2)
> I have not been able to reproduce a significant regression yet but will keep
> trying.
> 
> Does -machine pc-q35-rhel8.2.0 change the results with
> qemu-kvm-5.0.0-0.module+el8.3.0+7323+d54bb644.wrb200701.x86_64?
> 
> The pc-q35-rhel8.3.0 machine type includes changes to the virtio-blk queue
> size. This can affect the benchmark since it uses a high queue size.

Hi Stefan,

After changing machine type pc-q35-rhel8.2.0, no regression now.

http://kvm-perf.englab.nay.redhat.com/results/regression/block8.3.0_qemu5.0.0-0.scrmod-7066_kernel217round1/linux/xfs/diffmachien_type/qcow2.virtio_blk.*.x86_64.html
Comment 5 Stefan Hajnoczi 2020-08-06 15:35:02 UTC 
(In reply to Yanhui Ma from comment #4)
> (In reply to Stefan Hajnoczi from comment #2)
> > I have not been able to reproduce a significant regression yet but will keep
> > trying.
> > 
> > Does -machine pc-q35-rhel8.2.0 change the results with
> > qemu-kvm-5.0.0-0.module+el8.3.0+7323+d54bb644.wrb200701.x86_64?
> > 
> > The pc-q35-rhel8.3.0 machine type includes changes to the virtio-blk queue
> > size. This can affect the benchmark since it uses a high queue size.
> 
> Hi Stefan,
> 
> After changing machine type pc-q35-rhel8.2.0, no regression now.
> 
> http://kvm-perf.englab.nay.redhat.com/results/regression/block8.3.0_qemu5.0.
> 0-0.scrmod-7066_kernel217round1/linux/xfs/diffmachien_type/qcow2.virtio_blk.
> *.x86_64.html

Thank you for confirming.

I will investigate how the machine type change caused the regression.
Comment 6 Stefan Hajnoczi 2020-09-16 13:32:11 UTC 
I have asked the author of the patch that seems to have introduced the regression:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg734187.html
Comment 7 Stefan Hajnoczi 2020-09-29 14:16:20 UTC 
There is currently more activity on the POWER version of this bug. Mark as duplicate.

*** This bug has been marked as a duplicate of bug 1876455 ***
Comment 8 Yanhui Ma 2020-10-14 03:54:50 UTC 
Hi Stefan,

There are some differences between the bug and bug1876455. We test raw+blk, qcow2+blk, raw+scsi and qcow2+scsi these combinations for x86 and the regression only exists for raw+blk and qcow2+blk. However, only qcow2+scsi is tested for power and the regression exists for virtio-scsi. Besides, the test and bug verification for x86 and power are independent. Could I open the bug to track the test and bug verification on x86? When it is fixed, then close it.
Comment 9 Stefan Hajnoczi 2020-10-28 10:45:50 UTC 
(In reply to Yanhui Ma from comment #8)
> There are some differences between the bug and bug1876455. We test raw+blk,
> qcow2+blk, raw+scsi and qcow2+scsi these combinations for x86 and the
> regression only exists for raw+blk and qcow2+blk. However, only qcow2+scsi
> is tested for power and the regression exists for virtio-scsi. Besides, the
> test and bug verification for x86 and power are independent. Could I open
> the bug to track the test and bug verification on x86? When it is fixed,
> then close it.

Sure, if you would like to track them separately please go ahead.
Comment 13 Stefan Hajnoczi 2021-07-21 14:44:36 UTC 
Hi, this sounds similar to bz1930286 where increasing the Linux AIO queue size regressed performance.

Stefano: could you provide a qemu-kvm-5.0.0-0.module+el8.3.0+7323+d54bb644.wrb200701.x86_64 backport RPM with your Linux AIO batch size fix? Alternatively it may be possible to reproduce and test your backport with a more recent qemu-kvm RPM if Yanhui Ma wants to try that instead.
Comment 28 John Ferlan 2021-09-08 19:10:41 UTC 
Bulk update: Move RHEL-AV bugs to RHEL8
Comment 33 RHEL Program Management 2022-01-21 07:27:13 UTC 
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.
Comment 35 Tingting Mao 2022-07-12 02:33:20 UTC 
Tried to comprare between qemu-kvm-4.2.0-25.module+el8.2.1+6985+9fd9d514 and latest rhel9.1 with qemu-kvm-core-7.0.0-7.el9. There is still big degradation, especailly for write mode, and the biggest degression is ~60%. Check below link for more detialsN

http://kvm-perf.englab.nay.redhat.com/results/regression/xfs8.3.0_vs_9.1.0/02/qcow2.virtio_blk.*.x86_64.html
http://kvm-perf.englab.nay.redhat.com/results/regression/xfs8.3.0_vs_9.1.0/02/qcow2.virtio_scsi.*.x86_64.html
http://kvm-perf.englab.nay.redhat.com/results/regression/xfs8.3.0_vs_9.1.0/02/raw.virtio_blk.*.x86_64.html
http://kvm-perf.englab.nay.redhat.com/results/regression/xfs8.3.0_vs_9.1.0/02/raw.virtio_scsi.*.x86_64.html
Comment 37 Stefan Hajnoczi 2022-08-09 14:47:32 UTC 
(In reply to Tingting Mao from comment #35)
> http://kvm-perf.englab.nay.redhat.com/results/regression/xfs8.3.0_vs_9.1.0/
> 02/raw.virtio_blk.*.x86_64.html

Is it possible to keep the guest kernel the same in the comparison?

Please capture iostat and blktrace data for 10 seconds on the host while the benchmark runs:
- iostat -d -k -x /dev/DISK 1 10
- blktrace -d /dev/DISK -w 10

Comparing the iostat output may reveal an obvious difference in the I/O pattern on the host. The blktrace data can be analyzed in detail depending on the iostat output.

Thanks!
Comment 38 Stefan Hajnoczi 2022-08-09 14:50:41 UTC 
(The iostat and blktrace data is only needed for raw.virtio_blk sequential write bs=16k iodepth=1 threads=16 test. The performance of the other sequential write tests is probably limited by the same bottleneck so we don't need to investigate all of them.)
Comment 40 Tingting Mao 2022-09-28 10:20:30 UTC 
(In reply to Stefan Hajnoczi from comment #37)
> (In reply to Tingting Mao from comment #35)
> > http://kvm-perf.englab.nay.redhat.com/results/regression/xfs8.3.0_vs_9.1.0/
> > 02/raw.virtio_blk.*.x86_64.html
> 
> Is it possible to keep the guest kernel the same in the comparison?
Yes, updated. And there is still the big degradation, check below please.

http://kvm-perf.englab.nay.redhat.com/results/regression/bz1859048/0928/qcow2.virtio_blk.*.x86_64.html
http://kvm-perf.englab.nay.redhat.com/results/regression/bz1859048/0928/qcow2.virtio_scsi.*.x86_64.html
http://kvm-perf.englab.nay.redhat.com/results/regression/bz1859048/0928/raw.virtio_blk.*.x86_64.html
http://kvm-perf.englab.nay.redhat.com/results/regression/bz1859048/0928/raw.virtio_scsi.*.x86_64.html

> Please capture iostat and blktrace data for 10 seconds on the host while the
> benchmark runs:
> - iostat -d -k -x /dev/DISK 1 10
> - blktrace -d /dev/DISK -w 10
I have tried 3 times separately during the testing, check below please.
For handling https://bugzilla.redhat.com/show_bug.cgi?id=1859048 

Tested with:
Qemu-kvm-4.2.0-25.module+el8.2.1+6985+9fd9d514
Kernel-4.18.0-193.8.1.el8_2.x86_64 

1st time:
# iostat -d -k -x /dev/sdb 1 10
Linux 4.18.0-193.8.1.el8_2.x86_64 (ibm-x3650m5-08.lab.eng.pek2.redhat.com) 	09/23/2022 	_x86_64_	(40 CPU)

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb            118.50   74.91   3601.91   1472.55    31.15     4.69  20.82   5.89    7.94    9.78   1.61    30.40    19.66   0.10   1.95

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb           2091.00    0.00 434912.00      0.00   327.00     0.00  13.52   0.00   95.93    0.00 199.53   207.99     0.00   0.46  96.00

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb           2104.00    0.00 439008.00      0.00   341.00     0.00  13.95   0.00   96.39    0.00 201.63   208.65     0.00   0.46  96.90

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb           2084.00    0.00 435392.00      0.00   334.00     0.00  13.81   0.00   96.68    0.00 200.53   208.92     0.00   0.46  96.30

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb           2137.00    0.00 442528.00      0.00   323.00     0.00  13.13   0.00   94.15    0.00 200.09   207.08     0.00   0.46  97.40

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb           2091.00    0.00 437344.00      0.00   346.00     0.00  14.20   0.00   95.42    0.00 198.47   209.16     0.00   0.46  97.00

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb           2075.00    0.00 435360.00      0.00   346.00     0.00  14.29   0.00   94.50    0.00 195.02   209.81     0.00   0.46  95.10

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb           1962.00    0.00 415744.00      0.00   124.00     0.00   5.94   0.00  117.88    0.00 230.29   211.90     0.00   0.47  91.40

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb           1883.00    0.00 396544.00      0.00    98.00     0.00   4.95   0.00  132.28    0.00 248.18   210.59     0.00   0.47  88.50

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb           1877.00    5.00 398352.00     34.00   156.00     0.00   7.67   0.00  107.07    3.80 200.06   212.23     6.80   0.48  89.40

# blktrace -d /dev/sdb -w 10
=== sdb ===
  CPU  0:                    0 events,        0 KiB data
  CPU  1:                    0 events,        0 KiB data
  CPU  2:                    0 events,        0 KiB data
  CPU  3:                    0 events,        0 KiB data
  CPU  4:                    0 events,        0 KiB data
  CPU  5:                    1 events,        1 KiB data
  CPU  6:                    0 events,        0 KiB data
  CPU  7:                 6970 events,      327 KiB data
  CPU  8:                10940 events,      513 KiB data
  CPU  9:                21533 events,     1010 KiB data
  CPU 10:                11220 events,      526 KiB data
  CPU 11:                   38 events,        2 KiB data
  CPU 12:                52683 events,     2470 KiB data
  CPU 13:                  518 events,       25 KiB data
  CPU 14:                    0 events,        0 KiB data
  CPU 15:                    0 events,        0 KiB data
  CPU 16:                    0 events,        0 KiB data
  CPU 17:                    0 events,        0 KiB data
  CPU 18:                    0 events,        0 KiB data
  CPU 19:                10182 events,      478 KiB data
  CPU 20:                10923 events,      513 KiB data
  CPU 21:                  105 events,        5 KiB data
  CPU 22:                    0 events,        0 KiB data
  CPU 23:                    0 events,        0 KiB data
  CPU 24:                    0 events,        0 KiB data
  CPU 25:                    0 events,        0 KiB data
  CPU 26:                11227 events,      527 KiB data
  CPU 27:                 6326 events,      297 KiB data
  CPU 28:                    0 events,        0 KiB data
  CPU 29:                    0 events,        0 KiB data
  CPU 30:                 7331 events,      344 KiB data
  CPU 31:                    0 events,        0 KiB data
  CPU 32:                    0 events,        0 KiB data
  CPU 33:                    0 events,        0 KiB data
  CPU 34:                    0 events,        0 KiB data
  CPU 35:                    0 events,        0 KiB data
  CPU 36:                    0 events,        0 KiB data
  CPU 37:                    0 events,        0 KiB data
  CPU 38:                    0 events,        0 KiB data
  CPU 39:                    0 events,        0 KiB data
  Total:                149997 events (dropped 0),     7032 KiB data

2nd time:
# iostat -d -k -x /dev/sdb 1 10
Linux 4.18.0-193.8.1.el8_2.x86_64 (ibm-x3650m5-08.lab.eng.pek2.redhat.com) 	09/23/2022 	_x86_64_	(40 CPU)

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb            120.52  104.65   4133.86   2172.15    31.21    20.74  20.57  16.54    9.31    7.86   1.87    34.30    20.76   0.10   2.30

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb              0.00  529.00      0.00 102544.00     0.00     2.00   0.00   0.38    0.00  127.55  67.20     0.00   193.84   0.45  23.80

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb              0.00  627.00      0.00 120440.00     0.00     2.00   0.00   0.32    0.00  110.41  68.92     0.00   192.09   0.44  27.70

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb              0.00  674.00      0.00 127928.00     0.00     9.00   0.00   1.32    0.00  113.81  76.36     0.00   189.80   0.44  29.70

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb              0.00  608.00      0.00 121176.00     0.00     8.00   0.00   1.30    0.00  108.83  65.86     0.00   199.30   0.45  27.60

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb              0.00  656.00      0.00 126696.00     0.00     6.00   0.00   0.91    0.00  106.91  69.81     0.00   193.13   0.44  29.00

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb              0.00  590.00      0.00 116504.00     0.00     8.00   0.00   1.34    0.00   90.98  53.38     0.00   197.46   0.46  27.40

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb              0.00  342.00      0.00  65000.00     0.00     2.00   0.00   0.58    0.00  223.27  76.19     0.00   190.06   0.51  17.60

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb              0.00  450.00      0.00  86896.00     0.00     8.00   0.00   1.75    0.00  147.49  66.14     0.00   193.10   0.48  21.80

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb              0.00  510.00      0.00 101480.00     0.00     3.00   0.00   0.58    0.00  114.60  58.20     0.00   198.98   0.46  23.70

# blktrace -d /dev/sdb -w 10
=== sdb ===
  CPU  0:                 1013 events,       48 KiB data
  CPU  1:                 1037 events,       49 KiB data
  CPU  2:                 1073 events,       51 KiB data
  CPU  3:                 1044 events,       49 KiB data
  CPU  4:                 1087 events,       51 KiB data
  CPU  5:                 1109 events,       52 KiB data
  CPU  6:                  671 events,       32 KiB data
  CPU  7:                 1864 events,       88 KiB data
  CPU  8:                 4269 events,      201 KiB data
  CPU  9:                 2459 events,      116 KiB data
  CPU 10:                    0 events,        0 KiB data
  CPU 11:                    0 events,        0 KiB data
  CPU 12:                    0 events,        0 KiB data
  CPU 13:                    0 events,        0 KiB data
  CPU 14:                    0 events,        0 KiB data
  CPU 15:                    0 events,        0 KiB data
  CPU 16:                    0 events,        0 KiB data
  CPU 17:                    0 events,        0 KiB data
  CPU 18:                    0 events,        0 KiB data
  CPU 19:                    0 events,        0 KiB data
  CPU 20:                 4097 events,      193 KiB data
  CPU 21:                 3265 events,      154 KiB data
  CPU 22:                 4916 events,      231 KiB data
  CPU 23:                 1224 events,       58 KiB data
  CPU 24:                 3322 events,      156 KiB data
  CPU 25:                  852 events,       40 KiB data
  CPU 26:                 3682 events,      173 KiB data
  CPU 27:                 1328 events,       63 KiB data
  CPU 28:                 1690 events,       80 KiB data
  CPU 29:                 1419 events,       67 KiB data
  CPU 30:                    0 events,        0 KiB data
  CPU 31:                    0 events,        0 KiB data
  CPU 32:                    0 events,        0 KiB data
  CPU 33:                    0 events,        0 KiB data
  CPU 34:                    0 events,        0 KiB data
  CPU 35:                    0 events,        0 KiB data
  CPU 36:                    0 events,        0 KiB data
  CPU 37:                    0 events,        0 KiB data
  CPU 38:                    0 events,        0 KiB data
  CPU 39:                    0 events,        0 KiB data
  Total:                 41421 events (dropped 0),     1943 KiB data

3rd time:
# iostat -d -k -x /dev/sdb 1 10
Linux 4.18.0-193.8.1.el8_2.x86_64 (ibm-x3650m5-08.lab.eng.pek2.redhat.com) 	09/23/2022 	_x86_64_	(40 CPU)

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb            405.68  300.40  12340.29   5912.91    69.49    37.60  14.62  11.12    9.02   12.88   7.24    30.42    19.68   0.10   7.34

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb           29648.00    0.00 120812.00      0.00   539.00     0.00   1.79   0.00    0.47    0.00   0.03     4.07     0.00   0.03 100.00

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb           29022.00    0.00 118368.00      0.00   553.00     0.00   1.87   0.00    0.48    0.00   0.06     4.08     0.00   0.03 100.00

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb           29069.00    0.00 119212.00      0.00   700.00     0.00   2.35   0.00    0.48    0.00   0.03     4.10     0.00   0.03 100.00

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb           29869.00    0.00 121028.00      0.00   371.00     0.00   1.23   0.00    0.47    0.00   0.03     4.05     0.00   0.03 100.00

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb           29857.00    0.00 121724.00      0.00   550.00     0.00   1.81   0.00    0.47    0.00   0.09     4.08     0.00   0.03  99.90

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb           30148.00    0.00 122860.00      0.00   541.00     0.00   1.76   0.00    0.46    0.00   0.07     4.08     0.00   0.03 100.00

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb           31300.00    0.00 128000.00      0.00   667.00     0.00   2.09   0.00    0.44    0.00   0.04     4.09     0.00   0.03 100.00

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb           31043.00    0.00 127052.00      0.00   689.00     0.00   2.17   0.00    0.45    0.00   0.01     4.09     0.00   0.03 100.00

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb           29759.00    5.00 121652.00     34.50   625.00     0.00   2.06   0.00    0.47    0.20   0.03     4.09     6.90   0.03 100.00

# blktrace -d /dev/sdb -w 10
=== sdb ===
  CPU  0:                89246 events,     4184 KiB data
  CPU  1:               163872 events,     7682 KiB data
  CPU  2:                98339 events,     4610 KiB data
  CPU  3:                83072 events,     3895 KiB data
  CPU  4:                43713 events,     2050 KiB data
  CPU  5:                21882 events,     1026 KiB data
  CPU  6:                11542 events,      542 KiB data
  CPU  7:                10948 events,      514 KiB data
  CPU  8:                10942 events,      513 KiB data
  CPU  9:                10938 events,      513 KiB data
  CPU 10:                    0 events,        0 KiB data
  CPU 11:                    0 events,        0 KiB data
  CPU 12:                32798 events,     1538 KiB data
  CPU 13:                32785 events,     1537 KiB data
  CPU 14:                33318 events,     1562 KiB data
  CPU 15:                43705 events,     2049 KiB data
  CPU 16:                21870 events,     1026 KiB data
  CPU 17:                10954 events,      514 KiB data
  CPU 18:               131089 events,     6145 KiB data
  CPU 19:               207540 events,     9729 KiB data
  CPU 20:                    0 events,        0 KiB data
  CPU 21:                    0 events,        0 KiB data
  CPU 22:                    0 events,        0 KiB data
  CPU 23:                    0 events,        0 KiB data
  CPU 24:                    0 events,        0 KiB data
  CPU 25:                    0 events,        0 KiB data
  CPU 26:                32783 events,     1537 KiB data
  CPU 27:                98327 events,     4610 KiB data
  CPU 28:               120162 events,     5633 KiB data
  CPU 29:                98337 events,     4610 KiB data
  CPU 30:                    0 events,        0 KiB data
  CPU 31:                    0 events,        0 KiB data
  CPU 32:               185692 events,     8705 KiB data
  CPU 33:               120164 events,     5633 KiB data
  CPU 34:                10962 events,      514 KiB data
  CPU 35:                10943 events,      513 KiB data
  CPU 36:                    0 events,        0 KiB data
  CPU 37:                 1250 events,       59 KiB data
  CPU 38:                10960 events,      514 KiB data
  CPU 39:                 3162 events,      149 KiB data
  Total:               1751295 events (dropped 0),    82093 KiB data



Tested with:
Qemu-kvm-7.1.0-1.el9
Kernel-5.14.0-167.el9.x86_64 

1st time:
#  iostat -d -k -x /dev/sdb 1 10
Linux 5.14.0-167.el9.x86_64 (ibm-x3650m5-08.lab.eng.pek2.redhat.com) 	09/27/2022 	_x86_64_	(40 CPU)

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb           3111.04  99477.23   995.22  24.24   11.07    31.98 2197.17  42550.85   297.61  11.93   19.50    19.37    0.11  13625.14     0.00   0.00  212.34 123401.48    0.62   14.77   10.78  87.34


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb              0.00      0.00     0.00   0.00    0.00     0.00  218.00  55808.00     4.00   1.80  910.53   256.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00  198.50 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb              0.00      0.00     0.00   0.00    0.00     0.00  204.00  52224.00     3.00   1.45  498.48   256.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00  101.69 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb              0.00      0.00     0.00   0.00    0.00     0.00  375.00  96000.00     8.00   2.09  307.81   256.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00  115.43 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb              0.00      0.00     0.00   0.00    0.00     0.00  260.00  66560.00     5.00   1.89  346.61   256.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   90.12 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb              0.00      0.00     0.00   0.00    0.00     0.00  249.00  63744.00     4.00   1.58  453.74   256.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00  112.98 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb              0.00      0.00     0.00   0.00    0.00     0.00  215.00  55040.00     4.00   1.83 1059.95   256.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00  227.89 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb              0.00      0.00     0.00   0.00    0.00     0.00  469.00 120064.00     9.00   1.88  240.71   256.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00  112.89 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb              0.00      0.00     0.00   0.00    0.00     0.00  492.00 125952.00    10.00   1.99  206.50   256.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00  101.60  99.80


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb              0.00      0.00     0.00   0.00    0.00     0.00  407.00 103172.00     8.00   1.93  252.24   253.49    0.00      0.00     0.00   0.00    0.00     0.00    4.00  194.50  103.44  99.40

# blktrace -d /dev/sdb -w 10
=== sdb ===
  CPU  0:                   26 events,        2 KiB data
  CPU  1:                 1521 events,       72 KiB data
  CPU  2:                 1417 events,       67 KiB data
  CPU  3:                 3212 events,      151 KiB data
  CPU  4:                 2056 events,       97 KiB data
  CPU  5:                 2341 events,      110 KiB data
  CPU  6:                 2105 events,       99 KiB data
  CPU  7:                 1173 events,       56 KiB data
  CPU  8:                  919 events,       44 KiB data
  CPU  9:                    0 events,        0 KiB data
  CPU 10:                    0 events,        0 KiB data
  CPU 11:                    0 events,        0 KiB data
  CPU 12:                    0 events,        0 KiB data
  CPU 13:                    0 events,        0 KiB data
  CPU 14:                    0 events,        0 KiB data
  CPU 15:                    0 events,        0 KiB data
  CPU 16:                    0 events,        0 KiB data
  CPU 17:                    0 events,        0 KiB data
  CPU 18:                    0 events,        0 KiB data
  CPU 19:                    0 events,        0 KiB data
  CPU 20:                    0 events,        0 KiB data
  CPU 21:                    0 events,        0 KiB data
  CPU 22:                    0 events,        0 KiB data
  CPU 23:                    0 events,        0 KiB data
  CPU 24:                    0 events,        0 KiB data
  CPU 25:                    0 events,        0 KiB data
  CPU 26:                    0 events,        0 KiB data
  CPU 27:                    0 events,        0 KiB data
  CPU 28:                    9 events,        1 KiB data
  CPU 29:                    0 events,        0 KiB data
  CPU 30:                    0 events,        0 KiB data
  CPU 31:                    0 events,        0 KiB data
  CPU 32:                    0 events,        0 KiB data
  CPU 33:                    0 events,        0 KiB data
  CPU 34:                    0 events,        0 KiB data
  CPU 35:                    1 events,        1 KiB data
  CPU 36:                    0 events,        0 KiB data
  CPU 37:                    0 events,        0 KiB data
  CPU 38:                    0 events,        0 KiB data
  CPU 39:                    0 events,        0 KiB data
  Total:                 14780 events (dropped 0),      693 KiB data

2nd time:
#  iostat -d -k -x /dev/sdb 1 10
Linux 5.14.0-167.el9.x86_64 (ibm-x3650m5-08.lab.eng.pek2.redhat.com) 	09/27/2022 	_x86_64_	(40 CPU)

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb           3124.64  99231.12   991.80  24.09   11.00    31.76 2190.30  42583.57   296.60  11.93   19.66    19.44    0.11  13578.09     0.00   0.00  212.34 123401.48    0.61   14.95   11.15  87.35


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb           22897.00  91784.00    48.00   0.21    5.50     4.01    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00  125.98 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb           22884.00  91676.00    35.00   0.15    5.51     4.01    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00  125.99 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb           22952.00  92016.00    51.00   0.22    5.50     4.01    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00  126.31 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb           23090.00  92484.00    31.00   0.13    5.47     4.01    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00  126.24 100.10


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb           22865.00  91680.00    55.00   0.24    5.52     4.01    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00  126.23 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb           22952.00  91992.00    45.00   0.20    5.50     4.01    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00  126.18 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb           23105.00  92628.00    53.00   0.23    5.46     4.01    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00  126.14 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb           22871.00  91724.00    58.00   0.25    5.51     4.01    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00  126.07 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb           22999.00  92136.00    35.00   0.15    5.48     4.01    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00  126.13 100.00

# blktrace -d /dev/sdb -w 10
=== sdb ===
  CPU  0:               284775 events,    13349 KiB data
  CPU  1:               358715 events,    16815 KiB data
  CPU  2:               185938 events,     8716 KiB data
  CPU  3:                11215 events,      526 KiB data
  CPU  4:                    0 events,        0 KiB data
  CPU  5:                11198 events,      525 KiB data
  CPU  6:                 5606 events,      263 KiB data
  CPU  7:                    0 events,        0 KiB data
  CPU  8:                    0 events,        0 KiB data
  CPU  9:                    0 events,        0 KiB data
  CPU 10:                   31 events,        2 KiB data
  CPU 11:                    7 events,        1 KiB data
  CPU 12:               185705 events,     8705 KiB data
  CPU 13:                94976 events,     4452 KiB data
  CPU 14:                    0 events,        0 KiB data
  CPU 15:               135043 events,     6331 KiB data
  CPU 16:                    0 events,        0 KiB data
  CPU 17:                    1 events,        1 KiB data
  CPU 18:                11156 events,      523 KiB data
  CPU 19:                    0 events,        0 KiB data
  CPU 20:                17847 events,      837 KiB data
  CPU 21:               218693 events,    10252 KiB data
  CPU 22:                11183 events,      525 KiB data
  CPU 23:                    0 events,        0 KiB data
  CPU 24:                11183 events,      525 KiB data
  CPU 25:                11179 events,      525 KiB data
  CPU 26:                    0 events,        0 KiB data
  CPU 27:                    0 events,        0 KiB data
  CPU 28:                    0 events,        0 KiB data
  CPU 29:                    0 events,        0 KiB data
  CPU 30:                    0 events,        0 KiB data
  CPU 31:                    0 events,        0 KiB data
  CPU 32:                    0 events,        0 KiB data
  CPU 33:                11161 events,      524 KiB data
  CPU 34:                    0 events,        0 KiB data
  CPU 35:                11174 events,      524 KiB data
  CPU 36:                11177 events,      524 KiB data
  CPU 37:                    0 events,        0 KiB data
  CPU 38:                    0 events,        0 KiB data
  CPU 39:                    0 events,        0 KiB data
  Total:               1587963 events (dropped 0),    74437 KiB data

3rd time:
# iostat -d -k -x /dev/sdb 1 10
Linux 5.14.0-167.el9.x86_64 (ibm-x3650m5-08.lab.eng.pek2.redhat.com) 	09/27/2022 	_x86_64_	(40 CPU)

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb           3148.08  99901.22   974.68  23.64   11.16    31.73 2204.87  42246.73   291.57  11.68   19.57    19.16    0.11  13328.87     0.00   0.00  212.34 123401.48    0.63   14.62   13.20  87.41


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb              0.00      0.00     0.00   0.00    0.00     0.00 1168.00  70976.00     0.00   0.00   14.19    60.77    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   16.57 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb              0.00      0.00     0.00   0.00    0.00     0.00 1081.00  65920.00     0.00   0.00   15.17    60.98    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   16.40 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb              0.00      0.00     0.00   0.00    0.00     0.00 1186.00  71040.00     0.00   0.00   14.07    59.90    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   16.69  99.90


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb              0.00      0.00     0.00   0.00    0.00     0.00 1242.00  75648.00     0.00   0.00   13.30    60.91    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   16.51 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb              0.00      0.00     0.00   0.00    0.00     0.00  993.00  60608.00     0.00   0.00   16.60    61.04    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   16.48 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb              0.00      0.00     0.00   0.00    0.00     0.00 1133.00  68992.00     0.00   0.00   14.50    60.89    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   16.43 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb              0.00      0.00     0.00   0.00    0.00     0.00 1123.00  68416.00     0.00   0.00   14.61    60.92    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   16.41 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb              0.00      0.00     0.00   0.00    0.00     0.00 1103.00  67648.00     0.00   0.00   14.94    61.33    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   16.48 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb              0.00      0.00     0.00   0.00    0.00     0.00 1190.00  71680.00     0.00   0.00   14.00    60.24    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   16.66 100.00

# blktrace -d /dev/sdb -w 10
=== sdb ===
  CPU  0:                 5675 events,      267 KiB data
  CPU  1:                 3509 events,      165 KiB data
  CPU  2:                 4964 events,      233 KiB data
  CPU  3:                 5665 events,      266 KiB data
  CPU  4:                 2769 events,      130 KiB data
  CPU  5:                13425 events,      630 KiB data
  CPU  6:                 6636 events,      312 KiB data
  CPU  7:                 4403 events,      207 KiB data
  CPU  8:                  233 events,       11 KiB data
  CPU  9:                 1909 events,       90 KiB data
  CPU 10:                    0 events,        0 KiB data
  CPU 11:                    0 events,        0 KiB data
  CPU 12:                    0 events,        0 KiB data
  CPU 13:                    1 events,        1 KiB data
  CPU 14:                    0 events,        0 KiB data
  CPU 15:                    0 events,        0 KiB data
  CPU 16:                    0 events,        0 KiB data
  CPU 17:                    0 events,        0 KiB data
  CPU 18:                    0 events,        0 KiB data
  CPU 19:                    0 events,        0 KiB data
  CPU 20:                 6032 events,      283 KiB data
  CPU 21:                 7629 events,      358 KiB data
  CPU 22:                  992 events,       47 KiB data
  CPU 23:                 1388 events,       66 KiB data
  CPU 24:                  301 events,       15 KiB data
  CPU 25:                  985 events,       47 KiB data
  CPU 26:                 1190 events,       56 KiB data
  CPU 27:                 6187 events,      291 KiB data
  CPU 28:                 5419 events,      255 KiB data
  CPU 29:                 1699 events,       80 KiB data
  CPU 30:                    0 events,        0 KiB data
  CPU 31:                    0 events,        0 KiB data
  CPU 32:                    0 events,        0 KiB data
  CPU 33:                    0 events,        0 KiB data
  CPU 34:                    0 events,        0 KiB data
  CPU 35:                    0 events,        0 KiB data
  CPU 36:                    0 events,        0 KiB data
  CPU 37:                    0 events,        0 KiB data
  CPU 38:                    0 events,        0 KiB data
  CPU 39:                    0 events,        0 KiB data
  Total:                 81011 events (dropped 0),     3798 KiB data
Comment 42 Stefan Hajnoczi 2022-10-11 14:22:33 UTC 
(In reply to Tingting Mao from comment #40)
> (In reply to Stefan Hajnoczi from comment #37)
> > (In reply to Tingting Mao from comment #35)
> > > http://kvm-perf.englab.nay.redhat.com/results/regression/xfs8.3.0_vs_9.1.0/
> > > 02/raw.virtio_blk.*.x86_64.html
> > 
> > Is it possible to keep the guest kernel the same in the comparison?
> Yes, updated. And there is still the big degradation, check below please.
> 
> http://kvm-perf.englab.nay.redhat.com/results/regression/bz1859048/0928/
> qcow2.virtio_blk.*.x86_64.html
> http://kvm-perf.englab.nay.redhat.com/results/regression/bz1859048/0928/
> qcow2.virtio_scsi.*.x86_64.html
> http://kvm-perf.englab.nay.redhat.com/results/regression/bz1859048/0928/raw.
> virtio_blk.*.x86_64.html
> http://kvm-perf.englab.nay.redhat.com/results/regression/bz1859048/0928/raw.
> virtio_scsi.*.x86_64.html
> 
> > Please capture iostat and blktrace data for 10 seconds on the host while the
> > benchmark runs:
> > - iostat -d -k -x /dev/DISK 1 10
> > - blktrace -d /dev/DISK -w 10
> I have tried 3 times separately during the testing, check below please.
> For handling https://bugzilla.redhat.com/show_bug.cgi?id=1859048 

Was this output collected by manually running iostat/blktrace while the benchmark suite was running? If yes, then I don't think it can be used for comparison because we don't know if the captured results correspond to the same benchmarks. I see that the iostat output sometimes shows reads and sometimes shows writes, so it seems the iostat results were captured across multiple benchmarks.

Let's focus just on sequential write bs=16k iodepth=1 threads=16. The other benchmarks should not be run so they don't affect the iostat/blktrace results.

Please gather blktrace output using "blkparse -i sdb.blktrace.0", where the filename argument is the <block-device>.blktrace.0 file produced by blktrace. If the output is too long, just 60 lines should be enough to understand the general pattern.

Thank you!
Comment 43 Tingting Mao 2022-10-26 10:22:35 UTC 
Tried with raw + virtio_blk only, and plus sequential write bs=16k iodepth=1 threads=16. Take look again please.

The CML for running case is:
# python3 ConfigTest.py --testcase=fio_perf.single_disk.file_system_block.localfs --guestname=RHEL.8.2.0 --vcpu=2 --vcpu_cores=1 --vcpu_threads=1 --vcpu_socket=2 --mem=4096 --clone=no --nrepeat=4 --machines=q35 --driveformat=virtio_blk --imageformat=raw


The compared result is: http://kvm-perf.englab.nay.redhat.com/results/regression/bz1859048/20221026/raw.virtio_blk.*.x86_64.html 


And the results of iostat and blktrace are:

Tested with:
qemu-kvm-7.1.0-1.el9
Kernel-5.14.0-167.el9.x86_64

Results:
# iostat -d -k -x /dev/sdb 1 10
Linux 5.14.0-167.el9.x86_64 (ibm-x3650m5-08.lab.eng.pek2.redhat.com) 	10/20/2022 	_x86_64_	(40 CPU)

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb            112.29   3550.08    34.23  23.36   11.08    31.61   80.12   1544.70    10.36  11.45   20.04    19.28    0.01    676.12     0.00   0.00  214.76 123401.48    0.02   15.33    0.70   3.17


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb              0.00      0.00     0.00   0.00    0.00     0.00 10844.00 177904.00     1.00   0.01    1.36    16.41    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   14.72 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb              0.00      0.00     0.00   0.00    0.00     0.00 7948.00 129152.00     0.00   0.00    1.88    16.25    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   14.95 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb              0.00      0.00     0.00   0.00    0.00     0.00 8957.00 146608.00    18.00   0.20    1.65    16.37    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   14.77 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb              0.00      0.00     0.00   0.00    0.00     0.00 12260.00 201024.00    27.00   0.22    1.20    16.40    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   14.74 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb              0.00      0.00     0.00   0.00    0.00     0.00 10051.00 163616.00    25.00   0.25    1.49    16.28    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   14.97 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb              0.00      0.00     0.00   0.00    0.00     0.00 9935.00 161184.00    16.00   0.16    1.51    16.22    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   14.99 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb              0.00      0.00     0.00   0.00    0.00     0.00 12538.00 208032.00   190.00   1.49    1.16    16.59    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   14.48 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb              0.00      0.00     0.00   0.00    0.00     0.00 9734.00 161072.00    82.00   0.84    1.51    16.55    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   14.69 100.00


Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sdb              0.00      0.00     0.00   0.00    0.00     0.00 7324.00 119488.00    34.00   0.46    2.05    16.31    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00   15.04 100.00

# blktrace -d /dev/sdb -w 10
=== sdb ===
  CPU  0:                20504 events,      962 KiB data
  CPU  1:                22361 events,     1049 KiB data
  CPU  2:                52102 events,     2443 KiB data
  CPU  3:                47424 events,     2224 KiB data
  CPU  4:                14760 events,      692 KiB data
  CPU  5:                23327 events,     1094 KiB data
  CPU  6:                 3563 events,      168 KiB data
  CPU  7:                10932 events,      513 KiB data
  CPU  8:                10941 events,      513 KiB data
  CPU  9:                 5973 events,      280 KiB data
  CPU 10:                   74 events,        4 KiB data
  CPU 11:                  302 events,       15 KiB data
  CPU 12:                21967 events,     1030 KiB data
  CPU 13:                28365 events,     1330 KiB data
  CPU 14:                31933 events,     1497 KiB data
  CPU 15:                10883 events,      511 KiB data
  CPU 16:                16012 events,      751 KiB data
  CPU 17:                 7755 events,      364 KiB data
  CPU 18:                 8034 events,      377 KiB data
  CPU 19:                 9246 events,      434 KiB data
  CPU 20:                10943 events,      513 KiB data
  CPU 21:                  202 events,       10 KiB data
  CPU 22:                 4866 events,      229 KiB data
  CPU 23:                 5095 events,      239 KiB data
  CPU 24:                 6392 events,      300 KiB data
  CPU 25:                  108 events,        6 KiB data
  CPU 26:                  789 events,       38 KiB data
  CPU 27:                 4702 events,      221 KiB data
  CPU 28:                10944 events,      514 KiB data
  CPU 29:                10940 events,      513 KiB data
  CPU 30:                 1375 events,       65 KiB data
  CPU 31:                 1483 events,       70 KiB data
  CPU 32:                    1 events,        1 KiB data
  CPU 33:                    0 events,        0 KiB data
  CPU 34:                    0 events,        0 KiB data
  CPU 35:                    0 events,        0 KiB data
  CPU 36:                 3438 events,      162 KiB data
  CPU 37:                 1101 events,       52 KiB data
  CPU 38:                 1504 events,       71 KiB data
  CPU 39:                    0 events,        0 KiB data
  Total:                410341 events (dropped 0),    19236 KiB data

# blkparse -i sdb.blktrace.0
   8,16   5        1     0.000000000     0  C  WS 341472 + 32 [0]
  8,16   5        2     0.000008773     0  C  WS 340128 + 32 [0]
  8,16   5        3     0.000248152 962435  Q  WS 340160 + 32 [qemu-kvm]
  8,16   5        4     0.000251605 962435  G  WS 340160 + 32 [qemu-kvm]
  8,16   5        5     0.000252151 962435  P   N [qemu-kvm]
  8,16   5        6     0.000252975 962435  U   N [qemu-kvm] 1
  8,16   5        7     0.000253811 962435  I  WS 340160 + 32 [qemu-kvm]
  8,16   5        8     0.000259765 962435  D  WS 340160 + 32 [qemu-kvm]
  8,16   5        9     0.000266745 962435  Q  WS 341504 + 32 [qemu-kvm]
  8,16   5       10     0.000267695 962435  G  WS 341504 + 32 [qemu-kvm]
  8,16   5       11     0.000267915 962435  P   N [qemu-kvm]
  8,16   5       12     0.000268158 962435  U   N [qemu-kvm] 1
  8,16   5       13     0.000268451 962435  I  WS 341504 + 32 [qemu-kvm]
  8,16   5       14     0.000270085 962435  D  WS 341504 + 32 [qemu-kvm]
  8,16   5       15     0.003982349     0  C  WS 341664 + 32 [0]
  8,16   5       16     0.003986229     0  C  WS 341408 + 32 [0]
  8,16   5       17     0.004141823 962435  Q  WS 341696 + 32 [qemu-kvm]
  8,16   5       18     0.004143424 962435  G  WS 341696 + 32 [qemu-kvm]
  8,16   5       19     0.004143646 962435  P   N [qemu-kvm]
  8,16   5       20     0.004144008 962435  U   N [qemu-kvm] 1
  8,16   5       21     0.004144388 962435  I  WS 341696 + 32 [qemu-kvm]
  8,16   5       22     0.004146762 962435  D  WS 341696 + 32 [qemu-kvm]
  8,16   5       23     0.004153590 962435  Q  WS 341440 + 32 [qemu-kvm]
  8,16   5       24     0.004154005 962435  G  WS 341440 + 32 [qemu-kvm]
  8,16   5       25     0.004154153 962435  P   N [qemu-kvm]
  8,16   5       26     0.004154308 962435  U   N [qemu-kvm] 1
  8,16   5       27     0.004154429 962435  I  WS 341440 + 32 [qemu-kvm]
  8,16   5       28     0.004155151 962435  D  WS 341440 + 32 [qemu-kvm]
  8,16   5       29     0.005107958     0  C  WS 340928 + 32 [0]
  8,16   5       30     0.005116731     0  C  WS 341216 + 32 [0]
  8,16   5       31     0.005298080 962435  Q  WS 340960 + 32 [qemu-kvm]
  8,16   5       32     0.005299737 962435  G  WS 340960 + 32 [qemu-kvm]
  8,16   5       33     0.005299969 962435  P   N [qemu-kvm]
  8,16   5       34     0.005300331 962435  U   N [qemu-kvm] 1
  8,16   5       35     0.005300717 962435  I  WS 340960 + 32 [qemu-kvm]
  8,16   5       36     0.005303357 962435  D  WS 340960 + 32 [qemu-kvm]
  8,16   5       37     0.005319298 962435  Q  WS 341248 + 32 [qemu-kvm]
  8,16   5       38     0.005319808 962435  G  WS 341248 + 32 [qemu-kvm]
  8,16   5       39     0.005319955 962435  P   N [qemu-kvm]
  8,16   5       40     0.005320084 962435  U   N [qemu-kvm] 1
  8,16   5       41     0.005320256 962435  I  WS 341248 + 32 [qemu-kvm]
  8,16   5       42     0.005321274 962435  D  WS 341248 + 32 [qemu-kvm]
  8,16   5       43     0.005688280     0  C  WS 341536 + 32 [0]
  8,16   5       44     0.005696673     0  C  WS 341248 + 32 [0]
  8,16   5       45     0.005864785 962435  Q  WS 341280 + 32 [qemu-kvm]
  8,16   5       46     0.005868189 962435  G  WS 341280 + 32 [qemu-kvm]
  8,16   5       47     0.005868732 962435  P   N [qemu-kvm]
  8,16   5       48     0.005869562 962435  U   N [qemu-kvm] 1
  8,16   5       49     0.005870455 962435  I  WS 341280 + 32 [qemu-kvm]
  8,16   5       50     0.005875769 962435  D  WS 341280 + 32 [qemu-kvm]
  8,16   5       51     0.005882475 962435  Q  WS 341568 + 32 [qemu-kvm]
  8,16   5       52     0.005895545 962435  G  WS 341568 + 32 [qemu-kvm]
  8,16   5       53     0.005895842 962435  P   N [qemu-kvm]
  8,16   5       54     0.005896212 962435  U   N [qemu-kvm] 1
  8,16   5       55     0.005896532 962435  I  WS 341568 + 32 [qemu-kvm]
  8,16   5       56     0.005898332 962435  D  WS 341568 + 32 [qemu-kvm]
  8,16   5       57     0.006188346     0  C  WS 341152 + 32 [0]
  8,16   5       58     0.006190628     0  C  WS 341344 + 32 [0]
  8,16   5       59     0.006298851 962435  Q  WS 341184 + 32 [qemu-kvm]
  8,16   5       60     0.006299875 962435  G  WS 341184 + 32 [qemu-kvm]
  8,16   5       61     0.006300040 962435  P   N [qemu-kvm]
  8,16   5       62     0.006300266 962435  U   N [qemu-kvm] 1
….
…..
Total (sdb):
 Reads Queued:           0,        0KiB  Writes Queued:      57,819,  935,152KiB
 Read Dispatches:        0,        0KiB  Write Dispatches:   57,814,  935,152KiB
 Reads Requeued:         0               Writes Requeued:         0
 Reads Completed:        0,        0KiB  Writes Completed:   57,814,  935,152KiB
 Read Merges:            0,        0KiB  Write Merges:            5,       80KiB
 IO unplugs:        55,931               Timer unplugs:           0

Throughput (R/W): 0KiB/s / 93,928KiB/s
Events (sdb): 400,942 entries
Skips: 0 forward (0 -   0.0%)
Input file sdb.blktrace.0 added
Input file sdb.blktrace.1 added
Input file sdb.blktrace.2 added
Input file sdb.blktrace.3 added
Input file sdb.blktrace.4 added
Input file sdb.blktrace.5 added
Input file sdb.blktrace.6 added
Input file sdb.blktrace.7 added
Input file sdb.blktrace.8 added
Input file sdb.blktrace.9 added
Input file sdb.blktrace.10 added
Input file sdb.blktrace.11 added
Input file sdb.blktrace.12 added
Input file sdb.blktrace.13 added
Input file sdb.blktrace.14 added
Input file sdb.blktrace.15 added
Input file sdb.blktrace.16 added
Input file sdb.blktrace.17 added
Input file sdb.blktrace.18 added
Input file sdb.blktrace.19 added
Input file sdb.blktrace.20 added
Input file sdb.blktrace.21 added
Input file sdb.blktrace.22 added
Input file sdb.blktrace.23 added
Input file sdb.blktrace.24 added
Input file sdb.blktrace.25 added
Input file sdb.blktrace.26 added
Input file sdb.blktrace.27 added
Input file sdb.blktrace.28 added
Input file sdb.blktrace.29 added
Input file sdb.blktrace.30 added
Input file sdb.blktrace.31 added
Input file sdb.blktrace.32 added
Input file sdb.blktrace.36 added
Input file sdb.blktrace.37 added
Input file sdb.blktrace.38 added



Tested with:
qemu-kvm-4.2.0-25.module+el8.2.1+6985+9fd9d514
Kernel-4.18.0-193.8.1.el8_2.x86_64

Results:
# iostat -d -k -x /dev/sdb 1 10
Linux 4.18.0-193.8.1.el8_2.x86_64 (ibm-x3650m5-08.lab.eng.pek2.redhat.com) 	10/26/2022 	_x86_64_	(40 CPU)

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb              4.96   12.94     23.74    239.61     0.02     0.06   0.33   0.49    0.23    1.16   0.01     4.78    18.51   0.04   0.08

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb              0.00 22676.00      0.00 398496.00     0.00   107.00   0.00   0.47    0.00    0.58   1.35     0.00    17.57   0.04  90.80

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb              0.00 22590.00      0.00 392912.00     0.00     8.00   0.00   0.04    0.00    0.59   1.72     0.00    17.39   0.04  90.50

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb              0.00 22958.00      0.00 395584.00     0.00   121.00   0.00   0.52    0.00    0.59   1.62     0.00    17.23   0.04  90.90

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb              0.00 22179.00      0.00 393968.00     0.00   223.00   0.00   1.00    0.00    0.59   1.70     0.00    17.76   0.04  90.20

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb              0.00 22689.00      0.00 384288.00     0.00    62.00   0.00   0.27    0.00    0.61   1.99     0.00    16.94   0.04  90.60

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb              0.00 22482.00      0.00 403696.00     0.00   144.00   0.00   0.64    0.00    0.57   1.26     0.00    17.96   0.04  91.40

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb              0.00 21084.00      0.00 380096.00     0.00   174.00   0.00   0.82    0.00    0.61   2.04     0.00    18.03   0.04  90.50

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sdb              0.00 21115.00      0.00 378288.00     0.00   295.00   0.00   1.38    0.00    0.61   1.64     0.00    17.92   0.04  90.90

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util

# blktrace -d /dev/sdb -w 10
=== sdb ===
  CPU  0:                20358 events,      955 KiB data
  CPU  1:                 5380 events,      253 KiB data
  CPU  2:                17332 events,      813 KiB data
  CPU  3:                21432 events,     1005 KiB data
  CPU  4:                54961 events,     2577 KiB data
  CPU  5:                14699 events,      690 KiB data
  CPU  6:                12280 events,      576 KiB data
  CPU  7:                 5562 events,      261 KiB data
  CPU  8:                 6425 events,      302 KiB data
  CPU  9:                24330 events,     1141 KiB data
  CPU 10:                    0 events,        0 KiB data
  CPU 11:                    0 events,        0 KiB data
  CPU 12:                    0 events,        0 KiB data
  CPU 13:                    0 events,        0 KiB data
  CPU 14:                    0 events,        0 KiB data
  CPU 15:                    0 events,        0 KiB data
  CPU 16:                    0 events,        0 KiB data
  CPU 17:                    0 events,        0 KiB data
  CPU 18:                    0 events,        0 KiB data
  CPU 19:                    1 events,        1 KiB data
  CPU 20:                30155 events,     1414 KiB data
  CPU 21:                36040 events,     1690 KiB data
  CPU 22:                69878 events,     3276 KiB data
  CPU 23:                52070 events,     2441 KiB data
  CPU 24:                52852 events,     2478 KiB data
  CPU 25:                33486 events,     1570 KiB data
  CPU 26:                38022 events,     1783 KiB data
  CPU 27:                38272 events,     1795 KiB data
  CPU 28:                35395 events,     1660 KiB data
  CPU 29:                27334 events,     1282 KiB data
  CPU 30:                    0 events,        0 KiB data
  CPU 31:                    0 events,        0 KiB data
  CPU 32:                    0 events,        0 KiB data
  CPU 33:                    0 events,        0 KiB data
  CPU 34:                    0 events,        0 KiB data
  CPU 35:                    0 events,        0 KiB data
  CPU 36:                    0 events,        0 KiB data
  CPU 37:                    0 events,        0 KiB data
  CPU 38:                    0 events,        0 KiB data
  CPU 39:                    0 events,        0 KiB data
  Total:                596264 events (dropped 0),    27951 KiB data

# blkparse -i sdb.blktrace.0
  8,16  20        1     0.000000000     0  C  WS 671744 + 32 [0] 
  8,16  20        2     0.000005140     0  C  WS 671936 + 32 [0] 
  8,16  20        3     0.000101162 35311  Q  WS 671776 + 32 [qemu-kvm]
  8,16  20        4     0.000102816 35311  G  WS 671776 + 32 [qemu-kvm]
  8,16  20        5     0.000103316 35311  P   N [qemu-kvm]
  8,16  20        6     0.000105989 35311  Q  WS 671968 + 32 [qemu-kvm]
  8,16  20        7     0.000107059 35311  G  WS 671968 + 32 [qemu-kvm]
  8,16  20        8     0.000107799 35311  U   N [qemu-kvm] 2
  8,16  20        9     0.000108289 35311  I  WS 671776 + 32 [qemu-kvm]
  8,16  20       10     0.000108639 35311  I  WS 671968 + 32 [qemu-kvm]
  8,16  20       11     0.000110259 35311  D  WS 671776 + 32 [qemu-kvm]
  8,16  20       12     0.000113379 35311  D  WS 671968 + 32 [qemu-kvm]
  8,16  20       13     0.000336848     0  C  WS 672000 + 32 [0] 
  8,16  20       14     0.000338119     0  C  WS 672032 + 32 [0] 
  8,16  20       15     0.000426438 35311  Q  WS 672032 + 32 [qemu-kvm]
  8,16  20       16     0.000427041 35311  G  WS 672032 + 32 [qemu-kvm]
  8,16  20       17     0.000427225 35311  P   N [qemu-kvm]
  8,16  20       18     0.000427522 35311  U   N [qemu-kvm] 1
  8,16  20       19     0.000427743 35311  I  WS 672032 + 32 [qemu-kvm]
  8,16  20       20     0.000428423 35311  D  WS 672032 + 32 [qemu-kvm]
  8,16  20       21     0.000438327 35311  Q  WS 672064 + 32 [qemu-kvm]
  8,16  20       22     0.000438681 35311  G  WS 672064 + 32 [qemu-kvm]
  8,16  20       23     0.000438873 35311  P   N [qemu-kvm]
  8,16  20       24     0.000439103 35311  U   N [qemu-kvm] 1
  8,16  20       25     0.000439287 35311  I  WS 672064 + 32 [qemu-kvm]
  8,16  20       26     0.000439761 35311  D  WS 672064 + 32 [qemu-kvm]
  8,16  20       27     0.000682131     0  C  WS 671744 + 32 [0] 
  8,16  20       28     0.000684831     0  C  WS 671936 + 32 [0] 
  8,16  20       29     0.000790277 35311  Q  WS 671968 + 32 [qemu-kvm]
  8,16  20       30     0.000790927 35311  G  WS 671968 + 32 [qemu-kvm]
  8,16  20       31     0.000791100 35311  P   N [qemu-kvm]
  8,16  20       32     0.000791401 35311  U   N [qemu-kvm] 1
  8,16  20       33     0.000791621 35311  I  WS 671968 + 32 [qemu-kvm]
  8,16  20       34     0.000792406 35311  D  WS 671968 + 32 [qemu-kvm]
  8,16  20       35     0.000876548 35311  Q  WS 671776 + 32 [qemu-kvm]
  8,16  20       36     0.000877068 35311  G  WS 671776 + 32 [qemu-kvm]
  8,16  20       37     0.000877243 35311  P   N [qemu-kvm]
  8,16  20       38     0.000877481 35311  U   N [qemu-kvm] 1
  8,16  20       39     0.000877677 35311  I  WS 671776 + 32 [qemu-kvm]
  8,16  20       40     0.000878141 35311  D  WS 671776 + 32 [qemu-kvm]
  8,16  20       41     0.001024035     0  C  WS 671872 + 32 [0] 
  8,16  20       42     0.001040140     0  C  WS 672192 + 32 [0] 
  8,16  20       43     0.001139662 35311  Q  WS 671904 + 32 [qemu-kvm]
  8,16  20       44     0.001140238 35311  G  WS 671904 + 32 [qemu-kvm]
  8,16  20       45     0.001140424 35311  P   N [qemu-kvm]
  8,16  20       46     0.001140786 35311  U   N [qemu-kvm] 1
  8,16  20       47     0.001140994 35311  I  WS 671904 + 32 [qemu-kvm]
  8,16  20       48     0.001141745 35311  D  WS 671904 + 32 [qemu-kvm]
  8,16  20       49     0.001178198 35311  Q  WS 672224 + 32 [qemu-kvm]
  8,16  20       50     0.001178908 35311  G  WS 672224 + 32 [qemu-kvm]
  8,16  20       51     0.001179078 35311  P   N [qemu-kvm]
  8,16  20       52     0.001179555 35311  U   N [qemu-kvm] 1
  8,16  20       53     0.001179762 35311  I  WS 672224 + 32 [qemu-kvm]
  8,16  20       54     0.001180596 35311  D  WS 672224 + 32 [qemu-kvm]
  8,16  20       55     0.001364301     0  C  WS 671968 + 32 [0] 
  8,16  20       56     0.001365783     0  C  WS 671744 + 32 [0] 
  8,16  20       57     0.001523709 35311  Q  WS 671776 + 32 [qemu-kvm]
  8,16  20       58     0.001524338 35311  G  WS 671776 + 32 [qemu-kvm]
  8,16  20       59     0.001524509 35311  P   N [qemu-kvm]
  8,16  20       60     0.001524858 35311  U   N [qemu-kvm] 1
  8,16  20       61     0.001524985 35311  I  WS 671776 + 32 [qemu-kvm]
  8,16  20       62     0.001525777 35311  D  WS 671776 + 32 [qemu-kvm]
……
……
                                                         
Total (sdb):
 Reads Queued:           0,        0KiB  Writes Queued:      90,217,    1,552MiB
 Read Dispatches:        0,        0KiB  Write Dispatches:   89,973,    1,552MiB
 Reads Requeued:         0               Writes Requeued:         0
 Reads Completed:        0,        0KiB  Writes Completed:   89,973,    1,552MiB
 Read Merges:            0,        0KiB  Write Merges:          243,    4,000KiB
 IO unplugs:        67,309               Timer unplugs:           0

Throughput (R/W): 0KiB/s / 155,209KiB/s
Events (sdb): 584,972 entries
Skips: 0 forward (0 -   0.0%)
Input file sdb.blktrace.0 added
Input file sdb.blktrace.1 added
Input file sdb.blktrace.2 added
Input file sdb.blktrace.3 added
Input file sdb.blktrace.4 added
Input file sdb.blktrace.5 added
Input file sdb.blktrace.6 added
Input file sdb.blktrace.7 added
Input file sdb.blktrace.8 added
Input file sdb.blktrace.9 added
Input file sdb.blktrace.19 added
Input file sdb.blktrace.20 added
Input file sdb.blktrace.21 added
Input file sdb.blktrace.22 added
Input file sdb.blktrace.23 added
Input file sdb.blktrace.24 added
Input file sdb.blktrace.25 added
Input file sdb.blktrace.26 added
Input file sdb.blktrace.27 added
Input file sdb.blktrace.28 added
Input file sdb.blktrace.29 added
Comment 48 RHEL Program Management 2023-02-21 07:27:49 UTC 
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.
Comment 51 Stefan Hajnoczi 2023-04-26 12:41:39 UTC 
Closing due to lack of resources to investigate.