Bug 1708300
Summary: | RFE: qemu-nbd vs NBD_FLAG_CAN_MULTI_CONN | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 9 | Reporter: | Eric Blake <eblake> |
Component: | qemu-kvm | Assignee: | Eric Blake <eblake> |
qemu-kvm sub component: | NBD | QA Contact: | Tingting Mao <timao> |
Status: | CLOSED ERRATA | Docs Contact: | |
Severity: | medium | ||
Priority: | medium | CC: | coli, eblake, jinzhao, juzhang, kkiwi, kwolf, mrezanin, rbalakri, rjones, virt-maint, zhencliu |
Version: | 9.0 | Keywords: | FutureFeature, Reopened, RFE, Triaged |
Target Milestone: | rc | ||
Target Release: | --- | ||
Hardware: | Unspecified | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | qemu-kvm-7.0.0-5.el9 | Doc Type: | If docs needed, set a value |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2022-11-15 09:53:23 UTC | Type: | Feature Request |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | qemu-7.0 |
Embargoed: |
Description
Eric Blake
2019-05-09 14:40:39 UTC
Server changes proposed at: https://lists.gnu.org/archive/html/qemu-devel/2019-08/msg02878.html QEMU has been recently split into sub-components and as a one-time operation to avoid breakage of tools, we are setting the QEMU sub-component of this BZ to "General". Please review and change the sub-component if necessary the next time you review this BZ. Thanks Commit 61cc87245 "nbd: Advertise multi-conn for shared read-only connections" landed in upstream qemu 4.2: It is now possible to test that a readonly export advertises the flag (qemu-nbd --list makes it easy). Qemu as client still does not take advantage of a multi-conn server, but no one seems to be pushing for that functionality at the moment. I've assigned medium severity / high-priority to this, as my understanding is that this became a feature commitment. Eric, can you update us on where this is at? Care to also check if the BZ headers / targeting are correct? (In reply to Klaus Heinrich Kiwi from comment #7) > I've assigned medium severity / high-priority to this, as my understanding > is that this became a feature commitment. Eric, can you update us on where > this is at? Care to also check if the BZ headers / targeting are correct? Discussed this one with Eric today * The draft patch is coded, but the testcases have not yet converged * There's some questions about whether the RFE is still relevant, given newer versions of v2v. Move RHEL-AV bugs to RHEL9. If necessary to resolve in RHEL8, then clone to the current RHEL8 release. Upstream patch proposed: https://lists.gnu.org/archive/html/qemu-devel/2021-08/msg04900.html but review stalled after: https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg00042.html I'd really love an impact analysis from Kevin, as qemu-storage-daemon will also be affected. Depending on whether we think qemu is always safe with regards to flushing a format driver, or only safe when the format driver is non-network, may impact whether we need a v2 patch. After evaluating this issue, there are no plans to address it further or fix it in an upcoming release. Therefore, it is being closed. If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened. I'm sorry, this bug was closed in error by an incorrect process that we have no control over. Reopening. (In reply to Eric Blake from comment #10) > Upstream patch proposed: > https://lists.gnu.org/archive/html/qemu-devel/2021-08/msg04900.html > > but review stalled after: > https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg00042.html > > I'd really love an impact analysis from Kevin, as qemu-storage-daemon will > also be affected. Depending on whether we think qemu is always safe with > regards to flushing a format driver, or only safe when the format driver is > non-network, may impact whether we need a v2 patch. Kevin, where you able to progress with this review? Please try to prioritize this as it has been more than a month without updates. Thanks Sorry, I missed that this was waiting for input from me. I've now commented upstream that I think it would be better to change the interface (tl;dr: QAPI should be a superset of the qemu-nbd command line, not just have some intersections). Given that the QAPI design is still under work upstream (to make sure QAPI can do everything qemu-nbd could), this now won't land until qemu 7.0. As for whether this needs backporting downstream, my initial thought is that this is more of a performance rather than a correctness feature, and if the measured performance increases aren't significant, the effort of backporting is less important. (In reply to Eric Blake from comment #15) > Given that the QAPI design is still under work upstream (to make sure QAPI > can do everything qemu-nbd could), this now won't land until qemu 7.0. As > for whether this needs backporting downstream, my initial thought is that > this is more of a performance rather than a correctness feature, and if the > measured performance increases aren't significant, the effort of backporting > is less important. Lowering the priority based on that. I guess this is realistically RHEL 7.1 material now Upstream v2 patch posted: https://lists.gnu.org/archive/html/qemu-devel/2022-02/msg03314.html Upstream v3 patch posted: https://lists.gnu.org/archive/html/qemu-devel/2022-03/msg03701.html Just fixing/changing to use DTM (Developer Target Milestone) of 12 instead. I see patches are still on list as of today, but this still gives room to be downstream by 23-May (DTM12). That will miss Comprehensive Test Cycle (CTC) 1. Hopefully changes can be downstream prior to DTM18 which is the cutoff to make CTC2. Upstream patches are merged; starting the downstream backport now. commit 58a6fdcc9efb2a7c1ef4893dca4aa5e8020ca3dc Author: Eric Blake <eblake> Date: Wed May 11 19:49:24 2022 -0500 nbd/server: Allow MULTI_CONN for shared writable exports According to the NBD spec, a server that advertises NBD_FLAG_CAN_MULTI_CONN promises that multiple client connections will not see any cache inconsistencies: when properly separated by a single flush, actions performed by one client will be visible to another client, regardless of which client did the flush. We always satisfy these conditions in qemu - even when we support multiple clients, ALL clients go through a single point of reference into the block layer, with no local caching. The effect of one client is instantly visible to the next client. Even if our backend were a network device, we argue that any multi-path caching effects that would cause inconsistencies in back-to-back actions not seeing the effect of previous actions would be a bug in that backend, and not the fault of caching in qemu. As such, it is safe to unconditionally advertise CAN_MULTI_CONN for any qemu NBD server situation that supports parallel clients. Note, however, that we don't want to advertise CAN_MULTI_CONN when we know that a second client cannot connect (for historical reasons, qemu-nbd defaults to a single connection while nbd-server-add and QMP commands default to unlimited connections; but we already have existing means to let either style of NBD server creation alter those defaults). This is visible by no longer advertising MULTI_CONN for 'qemu-nbd -r' without -e, as in the iotest nbd-qemu-allocation. The harder part of this patch is setting up an iotest to demonstrate behavior of multiple NBD clients to a single server. It might be possible with parallel qemu-io processes, but I found it easier to do in python with the help of libnbd, and help from Nir and Vladimir in writing the test. Signed-off-by: Eric Blake <eblake> Suggested-by: Nir Soffer <nsoffer> Suggested-by: Vladimir Sementsov-Ogievskiy <v.sementsov-og> Message-Id: <20220512004924.417153-3-eblake> Signed-off-by: Kevin Wolf <kwolf> QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass. Hi Eric, I have tried to verify this bug to compare between qemu-kvm-7.0.0-4.el9 and qemu-kvm-7.0.0-5.el9. The results indicate that: - For qemu-nbd --list, the result is the same between qemu-kvm-7.0.0-4.el9 and qemu-kvm-7.0.0-5.el9 - For performance, it indeed improved. - For read, IOPS=23.0k VS IOPS=24.0k - For write, IOPS=23.4k VS IOPS=24.5k - For randread, IOPS=20.1k VS IOPS=21.6k - For randwrite, IOPS=20.5k VS IOPS=22.1k Could you please help to check whether the result is okay? And need I to cover any other scenarios for this bug verification? Thanks. Steps: 1. Prepare the exported image and export it # qemu-img create -f qcow2 test.qcow2 4M # qemu-io -c 'w -P 1 0 2M' -c 'w -P 2 2M 2M' test.qcow2 -f qcow2 # qemu-nbd -f qcow2 -e 5 test.qcow2 -t 2. Boot up a guest with the image as a data disk /usr/libexec/qemu-kvm \ -S \ -name 'avocado-vt-vm1' \ -sandbox on \ -blockdev node-name=file_ovmf_code,driver=file,filename=/usr/share/OVMF/OVMF_CODE.secboot.fd,auto-read-only=on,discard=unmap \ -blockdev node-name=drive_ovmf_code,driver=raw,read-only=on,file=file_ovmf_code \ -blockdev node-name=file_ovmf_vars,driver=file,filename=/home/kvm_autotest_root/images/avocado-vt-vm1_rhel900-64-virtio-scsi.raw_VARS.fd,auto-read-only=on,discard=unmap \ -blockdev node-name=drive_ovmf_vars,driver=raw,read-only=off,file=file_ovmf_vars \ -machine q35,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \ -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \ -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 \ -nodefaults \ -device VGA,bus=pcie.0,addr=0x2 \ -m 4096 \ -object memory-backend-ram,size=4096M,id=mem-machine_mem \ -smp 2,maxcpus=2,cores=1,threads=1,dies=1,sockets=2 \ -cpu 'Broadwell',+kvm_pv_unhalt \ -chardev socket,server=on,wait=off,path=/tmp/avocado_w8c1hz6d/monitor-qmpmonitor1-20220530-023235-aTNBI6tR,id=qmp_id_qmpmonitor1 \ -mon chardev=qmp_id_qmpmonitor1,mode=control \ -chardev socket,server=on,wait=off,path=/tmp/avocado_w8c1hz6d/monitor-catch_monitor-20220530-023235-aTNBI6tR,id=qmp_id_catch_monitor \ -mon chardev=qmp_id_catch_monitor,mode=control \ -device pvpanic,ioport=0x505,id=idolO2nc \ -chardev socket,server=on,wait=off,path=/tmp/avocado_w8c1hz6d/serial-serial0-20220530-023235-aTNBI6tR,id=chardev_serial0 \ -device isa-serial,id=serial0,chardev=chardev_serial0 \ -chardev socket,id=seabioslog_id_20220530-023235-aTNBI6tR,path=/tmp/avocado_w8c1hz6d/seabios-20220530-023235-aTNBI6tR,server=on,wait=off \ -device isa-debugcon,chardev=seabioslog_id_20220530-023235-aTNBI6tR,iobase=0x402 \ -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \ -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \ -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \ -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/rhel900-64-virtio-scsi.raw,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_image1,driver=raw,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \ -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \ -blockdev node-name=file_disk1,driver=nbd,auto-read-only=on,discard=unmap,server.host=localhost,server.port=10809,server.type=inet,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_disk1,driver=raw,read-only=off,cache.direct=on,cache.no-flush=off,file=file_disk1 \ -device scsi-hd,id=disk1,drive=drive_disk1,write-cache=on \ -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \ -device virtio-net-pci,mac=9a:37:37:37:37:9e,id=id0COG1G,netdev=idRlnAZl,bus=pcie-root-port-3,addr=0x0 \ -netdev tap,id=idRlnAZl \ -vnc :0 \ -rtc base=utc,clock=host,driftfix=slew \ -boot menu=off,order=cdn,once=c,strict=off \ -enable-kvm \ -monitor stdio \ -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5 \ 3. Fio test in guest for the data disk Results: ======================================================================================================================================== In qemu-kvm-7.0.0-4.el9: # qemu-nbd --list exports available: 1 export: '' size: 4194304 flags: 0xced ( flush fua trim zeroes df cache fast-zero ) min block: 1 opt block: 4096 max block: 33554432 available meta contexts: 1 base:allocation # fio --rw=rw --bs=4k --runtime=1m --direct=1 --filename=/dev/sdb --name=job1 --ioengine=libaio --numjobs=16 --thread --group_reporting job1: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 ... fio-3.27 Starting 16 threads job1: (groupid=0, jobs=16): err= 0: pid=2234: Mon Jun 6 14:52:31 2022 read: IOPS=23.0k, BW=89.8MiB/s (94.2MB/s)(31.7MiB/353msec) slat (nsec): min=1796, max=41716k, avg=30188.32, stdev=1028538.06 clat (usec): min=39, max=41698, avg=232.67, stdev=664.68 lat (usec): min=55, max=42441, avg=262.97, stdev=1230.44 clat percentiles (usec): | 1.00th=[ 86], 5.00th=[ 143], 10.00th=[ 157], 20.00th=[ 172], | 30.00th=[ 184], 40.00th=[ 198], 50.00th=[ 208], 60.00th=[ 221], | 70.00th=[ 233], 80.00th=[ 247], 90.00th=[ 277], 95.00th=[ 322], | 99.00th=[ 660], 99.50th=[ 758], 99.90th=[ 1418], 99.95th=[ 1614], | 99.99th=[41681] write: IOPS=23.4k, BW=91.5MiB/s (95.9MB/s)(32.3MiB/353msec); 0 zone resets slat (nsec): min=1943, max=41725k, avg=45362.30, stdev=1291621.67 clat (nsec): min=1083, max=41780k, avg=247541.04, stdev=914073.47 lat (usec): min=61, max=42498, avg=293.01, stdev=1592.60 clat percentiles (usec): | 1.00th=[ 89], 5.00th=[ 145], 10.00th=[ 159], 20.00th=[ 174], | 30.00th=[ 188], 40.00th=[ 202], 50.00th=[ 215], 60.00th=[ 225], | 70.00th=[ 237], 80.00th=[ 253], 90.00th=[ 281], 95.00th=[ 330], | 99.00th=[ 725], 99.50th=[ 848], 99.90th=[ 1434], 99.95th=[ 1647], | 99.99th=[41681] lat (usec) : 2=0.01%, 50=0.02%, 100=1.65%, 250=78.19%, 500=17.66% lat (usec) : 750=1.75%, 1000=0.45% lat (msec) : 2=0.23%, 10=0.01%, 50=0.04% cpu : usr=0.74%, sys=1.07%, ctx=16938, majf=0, minf=16 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=8115,8269,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=89.8MiB/s (94.2MB/s), 89.8MiB/s-89.8MiB/s (94.2MB/s-94.2MB/s), io=31.7MiB (33.2MB), run=353-353msec WRITE: bw=91.5MiB/s (95.9MB/s), 91.5MiB/s-91.5MiB/s (95.9MB/s-95.9MB/s), io=32.3MiB (33.9MB), run=353-353msec Disk stats (read/write): sdb: ios=3910/3908, merge=0/0, ticks=889/918, in_queue=1806, util=57.27% # fio --rw=randrw --bs=4k --runtime=1m --direct=1 --filename=/dev/sdb --name=job1 --ioengine=libaio --numjobs=16 --thread --group_reporting job1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 ... fio-3.27 Starting 16 threads job1: (groupid=0, jobs=16): err= 0: pid=2271: Mon Jun 6 14:53:02 2022 read: IOPS=20.1k, BW=78.7MiB/s (82.5MB/s)(31.7MiB/403msec) slat (nsec): min=1791, max=41098k, avg=44176.41, stdev=1276387.04 clat (usec): min=52, max=41114, avg=253.70, stdev=912.49 lat (usec): min=55, max=41539, avg=298.00, stdev=1580.45 clat percentiles (usec): | 1.00th=[ 104], 5.00th=[ 139], 10.00th=[ 155], 20.00th=[ 178], | 30.00th=[ 192], 40.00th=[ 204], 50.00th=[ 215], 60.00th=[ 225], | 70.00th=[ 239], 80.00th=[ 258], 90.00th=[ 302], 95.00th=[ 453], | 99.00th=[ 603], 99.50th=[ 717], 99.90th=[ 1287], 99.95th=[ 6390], | 99.99th=[41157] write: IOPS=20.5k, BW=80.2MiB/s (84.0MB/s)(32.3MiB/403msec); 0 zone resets slat (usec): min=2, max=40660, avg=34.06, stdev=1093.19 clat (usec): min=51, max=40585, avg=241.83, stdev=455.25 lat (usec): min=60, max=41733, avg=276.01, stdev=1198.36 clat percentiles (usec): | 1.00th=[ 106], 5.00th=[ 143], 10.00th=[ 161], 20.00th=[ 180], | 30.00th=[ 196], 40.00th=[ 206], 50.00th=[ 217], 60.00th=[ 227], | 70.00th=[ 241], 80.00th=[ 262], 90.00th=[ 314], 95.00th=[ 453], | 99.00th=[ 627], 99.50th=[ 783], 99.90th=[ 1205], 99.95th=[ 1303], | 99.99th=[40633] lat (usec) : 100=0.81%, 250=75.01%, 500=21.12%, 750=2.56%, 1000=0.21% lat (msec) : 2=0.24%, 10=0.01%, 50=0.03% cpu : usr=0.40%, sys=1.34%, ctx=16605, majf=0, minf=16 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=8115,8269,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=78.7MiB/s (82.5MB/s), 78.7MiB/s-78.7MiB/s (82.5MB/s-82.5MB/s), io=31.7MiB (33.2MB), run=403-403msec WRITE: bw=80.2MiB/s (84.0MB/s), 80.2MiB/s-80.2MiB/s (84.0MB/s-84.0MB/s), io=32.3MiB (33.9MB), run=403-403msec Disk stats (read/write): sdb: ios=8211/8262, merge=0/0, ticks=1955/1902, in_queue=3857, util=75.41% ================================================================================================================================ In qemu-kvm-7.0.0-5.el9: # qemu-nbd --list exports available: 1 export: '' size: 4194304 flags: 0xded ( flush fua trim zeroes df multi cache fast-zero ) min block: 1 opt block: 4096 max block: 33554432 available meta contexts: 1 base:allocation # fio --rw=rw --bs=4k --runtime=1m --direct=1 --filename=/dev/sdb --name=job1 --ioengine=libaio --numjobs=16 --thread --group_reporting job1: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 ... fio-3.27 Starting 16 threads job1: (groupid=0, jobs=16): err= 0: pid=2132: Mon Jun 6 15:07:18 2022 read: IOPS=24.0k, BW=93.8MiB/s (98.3MB/s)(31.7MiB/338msec) slat (nsec): min=1791, max=40958k, avg=34120.43, stdev=1112246.84 clat (usec): min=63, max=40936, avg=226.65, stdev=462.20 lat (usec): min=70, max=41662, avg=260.91, stdev=1213.13 clat percentiles (usec): | 1.00th=[ 106], 5.00th=[ 159], 10.00th=[ 174], 20.00th=[ 186], | 30.00th=[ 196], 40.00th=[ 204], 50.00th=[ 212], 60.00th=[ 221], | 70.00th=[ 229], 80.00th=[ 243], 90.00th=[ 265], 95.00th=[ 297], | 99.00th=[ 502], 99.50th=[ 660], 99.90th=[ 1467], 99.95th=[ 3163], | 99.99th=[41157] write: IOPS=24.5k, BW=95.6MiB/s (100MB/s)(32.3MiB/338msec); 0 zone resets slat (nsec): min=1883, max=40952k, avg=19011.89, stdev=779597.34 clat (usec): min=56, max=40927, avg=240.38, stdev=786.06 lat (usec): min=59, max=41599, avg=259.54, stdev=1111.91 clat percentiles (usec): | 1.00th=[ 119], 5.00th=[ 161], 10.00th=[ 174], 20.00th=[ 188], | 30.00th=[ 198], 40.00th=[ 206], 50.00th=[ 215], 60.00th=[ 223], | 70.00th=[ 233], 80.00th=[ 245], 90.00th=[ 265], 95.00th=[ 306], | 99.00th=[ 562], 99.50th=[ 676], 99.90th=[ 1745], 99.95th=[ 3228], | 99.99th=[41157] lat (usec) : 100=0.71%, 250=83.42%, 500=14.71%, 750=0.82%, 1000=0.16% lat (msec) : 2=0.10%, 4=0.05%, 10=0.01%, 50=0.02% cpu : usr=0.45%, sys=1.29%, ctx=16835, majf=1, minf=17 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=8115,8269,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=93.8MiB/s (98.3MB/s), 93.8MiB/s-93.8MiB/s (98.3MB/s-98.3MB/s), io=31.7MiB (33.2MB), run=338-338msec WRITE: bw=95.6MiB/s (100MB/s), 95.6MiB/s-95.6MiB/s (100MB/s-100MB/s), io=32.3MiB (33.9MB), run=338-338msec Disk stats (read/write): sdb: ios=4919/4898, merge=0/0, ticks=1092/1100, in_queue=2192, util=62.10% # fio --rw=randrw --bs=4k --runtime=1m --direct=1 --filename=/dev/sdb --name=job1 --ioengine=libaio --numjobs=16 --thread --group_reporting job1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1 ... fio-3.27 Starting 16 threads job1: (groupid=0, jobs=16): err= 0: pid=2200: Mon Jun 6 15:07:55 2022 read: IOPS=21.6k, BW=84.5MiB/s (88.6MB/s)(31.7MiB/375msec) slat (nsec): min=1819, max=41118k, avg=35205.76, stdev=1116543.78 clat (usec): min=29, max=41041, avg=282.38, stdev=911.00 lat (usec): min=58, max=41810, avg=317.77, stdev=1447.16 clat percentiles (usec): | 1.00th=[ 106], 5.00th=[ 165], 10.00th=[ 184], 20.00th=[ 204], | 30.00th=[ 223], 40.00th=[ 239], 50.00th=[ 253], 60.00th=[ 265], | 70.00th=[ 277], 80.00th=[ 297], 90.00th=[ 326], 95.00th=[ 388], | 99.00th=[ 619], 99.50th=[ 725], 99.90th=[ 2114], 99.95th=[ 2278], | 99.99th=[41157] write: IOPS=22.1k, BW=86.1MiB/s (90.3MB/s)(32.3MiB/375msec); 0 zone resets slat (nsec): min=1947, max=41105k, avg=30342.79, stdev=1009523.52 clat (usec): min=2, max=2263, avg=267.68, stdev=119.79 lat (usec): min=58, max=41773, avg=298.21, stdev=1025.24 clat percentiles (usec): | 1.00th=[ 109], 5.00th=[ 169], 10.00th=[ 186], 20.00th=[ 206], | 30.00th=[ 225], 40.00th=[ 243], 50.00th=[ 255], 60.00th=[ 269], | 70.00th=[ 281], 80.00th=[ 297], 90.00th=[ 334], 95.00th=[ 420], | 99.00th=[ 635], 99.50th=[ 775], 99.90th=[ 2114], 99.95th=[ 2212], | 99.99th=[ 2278] lat (usec) : 4=0.01%, 50=0.01%, 100=0.71%, 250=45.87%, 500=50.12% lat (usec) : 750=2.78%, 1000=0.23% lat (msec) : 2=0.15%, 4=0.10%, 50=0.02% cpu : usr=0.51%, sys=1.54%, ctx=16625, majf=0, minf=16 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=8115,8269,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=84.5MiB/s (88.6MB/s), 84.5MiB/s-84.5MiB/s (88.6MB/s-88.6MB/s), io=31.7MiB (33.2MB), run=375-375msec WRITE: bw=86.1MiB/s (90.3MB/s), 86.1MiB/s-86.1MiB/s (90.3MB/s-90.3MB/s), io=32.3MiB (33.9MB), run=375-375msec Disk stats (read/write): sdb: ios=3859/3837, merge=0/0, ticks=961/971, in_queue=1931, util=60.17% (In reply to Tingting Mao from comment #25) > Hi Eric, > > I have tried to verify this bug to compare between qemu-kvm-7.0.0-4.el9 and > qemu-kvm-7.0.0-5.el9. The results indicate that: > - For qemu-nbd --list, the result is the same between qemu-kvm-7.0.0-4.el9 > and qemu-kvm-7.0.0-5.el9 Not quite. Notice below... > - For performance, it indeed improved. > - For read, IOPS=23.0k VS IOPS=24.0k > - For write, IOPS=23.4k VS IOPS=24.5k > - For randread, IOPS=20.1k VS IOPS=21.6k > - For randwrite, IOPS=20.5k VS IOPS=22.1k Good to hear! > > Could you please help to check whether the result is okay? And need I to > cover any other scenarios for this bug verification? > > Results: > ============================================================================= > =========================================================== > In qemu-kvm-7.0.0-4.el9: > # qemu-nbd --list > exports available: 1 > export: '' > size: 4194304 > flags: 0xced ( flush fua trim zeroes df cache fast-zero ) > =================================================== > In qemu-kvm-7.0.0-5.el9: > # qemu-nbd --list > exports available: 1 > export: '' > size: 4194304 > flags: 0xded ( flush fua trim zeroes df multi cache fast-zero ) The addition of the 'multi' flag is what we wanted to see! The improved performance is also nice. But you have indeed verified that the setting is advertised correctly. Thanks Eric, set this bug as verified accordingly. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: qemu-kvm security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:7967 |