Server changes proposed at:
https://lists.gnu.org/archive/html/qemu-devel/2019-08/msg02878.html

QEMU has been recently split into sub-components and as a one-time operation to avoid breakage of tools, we are setting the QEMU sub-component of this BZ to "General". Please review and change the sub-component if necessary the next time you review this BZ. Thanks

Commit 61cc87245 "nbd: Advertise multi-conn for shared read-only connections" landed in upstream qemu 4.2:

It is now possible to test that a readonly export advertises the flag (qemu-nbd --list makes it easy). Qemu as client still does not take advantage of a multi-conn server, but no one seems to be pushing for that functionality at the moment.

I've assigned medium severity / high-priority to this, as my understanding is that this became a feature commitment. Eric, can you update us on where this is at? Care to also check if the BZ headers / targeting are correct?

(In reply to Klaus Heinrich Kiwi from comment #7)
> I've assigned medium severity / high-priority to this, as my understanding
> is that this became a feature commitment. Eric, can you update us on where
> this is at? Care to also check if the BZ headers / targeting are correct?

Discussed this one with Eric today

 * The draft patch is coded, but the testcases have not yet converged
 * There's some questions about whether the RFE is still relevant, given newer versions of v2v.

Move RHEL-AV bugs to RHEL9. If necessary to resolve in RHEL8, then clone to the current RHEL8 release.

Upstream patch proposed:
https://lists.gnu.org/archive/html/qemu-devel/2021-08/msg04900.html

but review stalled after:
https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg00042.html

I'd really love an impact analysis from Kevin, as qemu-storage-daemon will also be affected.  Depending on whether we think qemu is always safe with regards to flushing a format driver, or only safe when the format driver is non-network, may impact whether we need a v2 patch.

After evaluating this issue, there are no plans to address it further or fix it in an upcoming release.  Therefore, it is being closed.  If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.

I'm sorry, this bug was closed in error by an incorrect process that
we have no control over.  Reopening.

(In reply to Eric Blake from comment #10)
> Upstream patch proposed:
> https://lists.gnu.org/archive/html/qemu-devel/2021-08/msg04900.html
> 
> but review stalled after:
> https://lists.gnu.org/archive/html/qemu-devel/2021-09/msg00042.html
> 
> I'd really love an impact analysis from Kevin, as qemu-storage-daemon will
> also be affected.  Depending on whether we think qemu is always safe with
> regards to flushing a format driver, or only safe when the format driver is
> non-network, may impact whether we need a v2 patch.

Kevin, where you able to progress with this review? Please try to prioritize this as it has been more than a month without updates.

Thanks

Sorry, I missed that this was waiting for input from me. I've now commented upstream that I think it would be better to change the interface (tl;dr: QAPI should be a superset of the qemu-nbd command line, not just have some intersections).

Given that the QAPI design is still under work upstream (to make sure QAPI can do everything qemu-nbd could), this now won't land until qemu 7.0.  As for whether this needs backporting downstream, my initial thought is that this is more of a performance rather than a correctness feature, and if the measured performance increases aren't significant, the effort of backporting is less important.

(In reply to Eric Blake from comment #15)
> Given that the QAPI design is still under work upstream (to make sure QAPI
> can do everything qemu-nbd could), this now won't land until qemu 7.0.  As
> for whether this needs backporting downstream, my initial thought is that
> this is more of a performance rather than a correctness feature, and if the
> measured performance increases aren't significant, the effort of backporting
> is less important.

Lowering the priority based on that. I guess this is realistically RHEL 7.1 material now

Upstream v2 patch posted:
https://lists.gnu.org/archive/html/qemu-devel/2022-02/msg03314.html

Upstream v3 patch posted:
https://lists.gnu.org/archive/html/qemu-devel/2022-03/msg03701.html

Just fixing/changing to use DTM (Developer Target Milestone) of 12 instead.  I see patches are still on list as of today, but this still gives room to be downstream by 23-May (DTM12). That will miss Comprehensive Test Cycle (CTC) 1. Hopefully changes can be downstream prior to DTM18 which is the cutoff to make CTC2.

Upstream patches are merged; starting the downstream backport now.

commit 58a6fdcc9efb2a7c1ef4893dca4aa5e8020ca3dc
Author: Eric Blake <eblake>
Date:   Wed May 11 19:49:24 2022 -0500

    nbd/server: Allow MULTI_CONN for shared writable exports
    
    According to the NBD spec, a server that advertises
    NBD_FLAG_CAN_MULTI_CONN promises that multiple client connections will
    not see any cache inconsistencies: when properly separated by a single
    flush, actions performed by one client will be visible to another
    client, regardless of which client did the flush.
    
    We always satisfy these conditions in qemu - even when we support
    multiple clients, ALL clients go through a single point of reference
    into the block layer, with no local caching.  The effect of one client
    is instantly visible to the next client.  Even if our backend were a
    network device, we argue that any multi-path caching effects that
    would cause inconsistencies in back-to-back actions not seeing the
    effect of previous actions would be a bug in that backend, and not the
    fault of caching in qemu.  As such, it is safe to unconditionally
    advertise CAN_MULTI_CONN for any qemu NBD server situation that
    supports parallel clients.
    
    Note, however, that we don't want to advertise CAN_MULTI_CONN when we
    know that a second client cannot connect (for historical reasons,
    qemu-nbd defaults to a single connection while nbd-server-add and QMP
    commands default to unlimited connections; but we already have
    existing means to let either style of NBD server creation alter those
    defaults).  This is visible by no longer advertising MULTI_CONN for
    'qemu-nbd -r' without -e, as in the iotest nbd-qemu-allocation.
    
    The harder part of this patch is setting up an iotest to demonstrate
    behavior of multiple NBD clients to a single server.  It might be
    possible with parallel qemu-io processes, but I found it easier to do
    in python with the help of libnbd, and help from Nir and Vladimir in
    writing the test.
    
    Signed-off-by: Eric Blake <eblake>
    Suggested-by: Nir Soffer <nsoffer>
    Suggested-by: Vladimir Sementsov-Ogievskiy <v.sementsov-og>
    Message-Id: <20220512004924.417153-3-eblake>
    Signed-off-by: Kevin Wolf <kwolf>

QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.

Hi Eric,

I have tried to verify this bug to compare between qemu-kvm-7.0.0-4.el9 and qemu-kvm-7.0.0-5.el9. The results indicate that:
 - For qemu-nbd --list, the result is the same between qemu-kvm-7.0.0-4.el9 and qemu-kvm-7.0.0-5.el9
 - For performance, it indeed improved.
   - For read, IOPS=23.0k VS IOPS=24.0k
   - For write, IOPS=23.4k VS IOPS=24.5k
   - For randread, IOPS=20.1k VS IOPS=21.6k
   - For randwrite, IOPS=20.5k VS IOPS=22.1k

Could you please help to check whether the result is okay? And need I to cover any other scenarios for this bug verification?

Thanks.


Steps:
1. Prepare the exported image and export it
# qemu-img create -f qcow2 test.qcow2 4M
# qemu-io -c 'w -P 1 0 2M' -c 'w -P 2 2M 2M' test.qcow2 -f qcow2
# qemu-nbd -f qcow2 -e 5 test.qcow2 -t

2. Boot up a guest with the image as a data disk
/usr/libexec/qemu-kvm \
    -S  \
    -name 'avocado-vt-vm1'  \
    -sandbox on  \
    -blockdev node-name=file_ovmf_code,driver=file,filename=/usr/share/OVMF/OVMF_CODE.secboot.fd,auto-read-only=on,discard=unmap \
    -blockdev node-name=drive_ovmf_code,driver=raw,read-only=on,file=file_ovmf_code \
    -blockdev node-name=file_ovmf_vars,driver=file,filename=/home/kvm_autotest_root/images/avocado-vt-vm1_rhel900-64-virtio-scsi.raw_VARS.fd,auto-read-only=on,discard=unmap \
    -blockdev node-name=drive_ovmf_vars,driver=raw,read-only=off,file=file_ovmf_vars \
    -machine q35,memory-backend=mem-machine_mem,pflash0=drive_ovmf_code,pflash1=drive_ovmf_vars \
    -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \
    -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0  \
    -nodefaults \
    -device VGA,bus=pcie.0,addr=0x2 \
    -m 4096 \
    -object memory-backend-ram,size=4096M,id=mem-machine_mem  \
    -smp 2,maxcpus=2,cores=1,threads=1,dies=1,sockets=2  \
    -cpu 'Broadwell',+kvm_pv_unhalt \
    -chardev socket,server=on,wait=off,path=/tmp/avocado_w8c1hz6d/monitor-qmpmonitor1-20220530-023235-aTNBI6tR,id=qmp_id_qmpmonitor1  \
    -mon chardev=qmp_id_qmpmonitor1,mode=control \
    -chardev socket,server=on,wait=off,path=/tmp/avocado_w8c1hz6d/monitor-catch_monitor-20220530-023235-aTNBI6tR,id=qmp_id_catch_monitor  \
    -mon chardev=qmp_id_catch_monitor,mode=control \
    -device pvpanic,ioport=0x505,id=idolO2nc \
    -chardev socket,server=on,wait=off,path=/tmp/avocado_w8c1hz6d/serial-serial0-20220530-023235-aTNBI6tR,id=chardev_serial0 \
    -device isa-serial,id=serial0,chardev=chardev_serial0  \
    -chardev socket,id=seabioslog_id_20220530-023235-aTNBI6tR,path=/tmp/avocado_w8c1hz6d/seabios-20220530-023235-aTNBI6tR,server=on,wait=off \
    -device isa-debugcon,chardev=seabioslog_id_20220530-023235-aTNBI6tR,iobase=0x402 \
    -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \
    -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
    -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \
    -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \
    -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=native,filename=/home/kvm_autotest_root/images/rhel900-64-virtio-scsi.raw,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_image1,driver=raw,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \
    -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \
    -blockdev node-name=file_disk1,driver=nbd,auto-read-only=on,discard=unmap,server.host=localhost,server.port=10809,server.type=inet,cache.direct=on,cache.no-flush=off \
    -blockdev node-name=drive_disk1,driver=raw,read-only=off,cache.direct=on,cache.no-flush=off,file=file_disk1 \
    -device scsi-hd,id=disk1,drive=drive_disk1,write-cache=on \
    -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \
    -device virtio-net-pci,mac=9a:37:37:37:37:9e,id=id0COG1G,netdev=idRlnAZl,bus=pcie-root-port-3,addr=0x0  \
    -netdev tap,id=idRlnAZl  \
    -vnc :0  \
    -rtc base=utc,clock=host,driftfix=slew  \
    -boot menu=off,order=cdn,once=c,strict=off \
    -enable-kvm \
    -monitor stdio \
    -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=5 \                                                                                                      

3. Fio test in guest for the data disk


Results:
========================================================================================================================================
In qemu-kvm-7.0.0-4.el9:
# qemu-nbd --list 
exports available: 1
 export: ''
  size:  4194304
  flags: 0xced ( flush fua trim zeroes df cache fast-zero )
  min block: 1
  opt block: 4096
  max block: 33554432
  available meta contexts: 1
   base:allocation

# fio --rw=rw --bs=4k --runtime=1m --direct=1 --filename=/dev/sdb --name=job1 --ioengine=libaio --numjobs=16 --thread --group_reporting
job1: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.27
Starting 16 threads

job1: (groupid=0, jobs=16): err= 0: pid=2234: Mon Jun  6 14:52:31 2022
  read: IOPS=23.0k, BW=89.8MiB/s (94.2MB/s)(31.7MiB/353msec)
    slat (nsec): min=1796, max=41716k, avg=30188.32, stdev=1028538.06
    clat (usec): min=39, max=41698, avg=232.67, stdev=664.68
     lat (usec): min=55, max=42441, avg=262.97, stdev=1230.44
    clat percentiles (usec):
     |  1.00th=[   86],  5.00th=[  143], 10.00th=[  157], 20.00th=[  172],
     | 30.00th=[  184], 40.00th=[  198], 50.00th=[  208], 60.00th=[  221],
     | 70.00th=[  233], 80.00th=[  247], 90.00th=[  277], 95.00th=[  322],
     | 99.00th=[  660], 99.50th=[  758], 99.90th=[ 1418], 99.95th=[ 1614],
     | 99.99th=[41681]
  write: IOPS=23.4k, BW=91.5MiB/s (95.9MB/s)(32.3MiB/353msec); 0 zone resets
    slat (nsec): min=1943, max=41725k, avg=45362.30, stdev=1291621.67
    clat (nsec): min=1083, max=41780k, avg=247541.04, stdev=914073.47
     lat (usec): min=61, max=42498, avg=293.01, stdev=1592.60
    clat percentiles (usec):
     |  1.00th=[   89],  5.00th=[  145], 10.00th=[  159], 20.00th=[  174],
     | 30.00th=[  188], 40.00th=[  202], 50.00th=[  215], 60.00th=[  225],
     | 70.00th=[  237], 80.00th=[  253], 90.00th=[  281], 95.00th=[  330],
     | 99.00th=[  725], 99.50th=[  848], 99.90th=[ 1434], 99.95th=[ 1647],
     | 99.99th=[41681]
  lat (usec)   : 2=0.01%, 50=0.02%, 100=1.65%, 250=78.19%, 500=17.66%
  lat (usec)   : 750=1.75%, 1000=0.45%
  lat (msec)   : 2=0.23%, 10=0.01%, 50=0.04%
  cpu          : usr=0.74%, sys=1.07%, ctx=16938, majf=0, minf=16
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=8115,8269,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=89.8MiB/s (94.2MB/s), 89.8MiB/s-89.8MiB/s (94.2MB/s-94.2MB/s), io=31.7MiB (33.2MB), run=353-353msec
  WRITE: bw=91.5MiB/s (95.9MB/s), 91.5MiB/s-91.5MiB/s (95.9MB/s-95.9MB/s), io=32.3MiB (33.9MB), run=353-353msec

Disk stats (read/write):
  sdb: ios=3910/3908, merge=0/0, ticks=889/918, in_queue=1806, util=57.27%


# fio --rw=randrw --bs=4k --runtime=1m --direct=1 --filename=/dev/sdb --name=job1 --ioengine=libaio --numjobs=16 --thread --group_reporting
job1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.27
Starting 16 threads

job1: (groupid=0, jobs=16): err= 0: pid=2271: Mon Jun  6 14:53:02 2022
  read: IOPS=20.1k, BW=78.7MiB/s (82.5MB/s)(31.7MiB/403msec)
    slat (nsec): min=1791, max=41098k, avg=44176.41, stdev=1276387.04
    clat (usec): min=52, max=41114, avg=253.70, stdev=912.49
     lat (usec): min=55, max=41539, avg=298.00, stdev=1580.45
    clat percentiles (usec):
     |  1.00th=[  104],  5.00th=[  139], 10.00th=[  155], 20.00th=[  178],
     | 30.00th=[  192], 40.00th=[  204], 50.00th=[  215], 60.00th=[  225],
     | 70.00th=[  239], 80.00th=[  258], 90.00th=[  302], 95.00th=[  453],
     | 99.00th=[  603], 99.50th=[  717], 99.90th=[ 1287], 99.95th=[ 6390],
     | 99.99th=[41157]
  write: IOPS=20.5k, BW=80.2MiB/s (84.0MB/s)(32.3MiB/403msec); 0 zone resets
    slat (usec): min=2, max=40660, avg=34.06, stdev=1093.19
    clat (usec): min=51, max=40585, avg=241.83, stdev=455.25
     lat (usec): min=60, max=41733, avg=276.01, stdev=1198.36
    clat percentiles (usec):
     |  1.00th=[  106],  5.00th=[  143], 10.00th=[  161], 20.00th=[  180],
     | 30.00th=[  196], 40.00th=[  206], 50.00th=[  217], 60.00th=[  227],
     | 70.00th=[  241], 80.00th=[  262], 90.00th=[  314], 95.00th=[  453],
     | 99.00th=[  627], 99.50th=[  783], 99.90th=[ 1205], 99.95th=[ 1303],
     | 99.99th=[40633]
  lat (usec)   : 100=0.81%, 250=75.01%, 500=21.12%, 750=2.56%, 1000=0.21%
  lat (msec)   : 2=0.24%, 10=0.01%, 50=0.03%
  cpu          : usr=0.40%, sys=1.34%, ctx=16605, majf=0, minf=16
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=8115,8269,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=78.7MiB/s (82.5MB/s), 78.7MiB/s-78.7MiB/s (82.5MB/s-82.5MB/s), io=31.7MiB (33.2MB), run=403-403msec
  WRITE: bw=80.2MiB/s (84.0MB/s), 80.2MiB/s-80.2MiB/s (84.0MB/s-84.0MB/s), io=32.3MiB (33.9MB), run=403-403msec

Disk stats (read/write):
  sdb: ios=8211/8262, merge=0/0, ticks=1955/1902, in_queue=3857, util=75.41%


================================================================================================================================
In qemu-kvm-7.0.0-5.el9:
# qemu-nbd --list
exports available: 1
 export: ''
  size:  4194304
  flags: 0xded ( flush fua trim zeroes df multi cache fast-zero )
  min block: 1
  opt block: 4096
  max block: 33554432
  available meta contexts: 1
   base:allocation

#  fio --rw=rw --bs=4k --runtime=1m --direct=1 --filename=/dev/sdb --name=job1 --ioengine=libaio --numjobs=16 --thread --group_reporting
job1: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.27
Starting 16 threads

job1: (groupid=0, jobs=16): err= 0: pid=2132: Mon Jun  6 15:07:18 2022
  read: IOPS=24.0k, BW=93.8MiB/s (98.3MB/s)(31.7MiB/338msec)
    slat (nsec): min=1791, max=40958k, avg=34120.43, stdev=1112246.84
    clat (usec): min=63, max=40936, avg=226.65, stdev=462.20
     lat (usec): min=70, max=41662, avg=260.91, stdev=1213.13
    clat percentiles (usec):
     |  1.00th=[  106],  5.00th=[  159], 10.00th=[  174], 20.00th=[  186],
     | 30.00th=[  196], 40.00th=[  204], 50.00th=[  212], 60.00th=[  221],
     | 70.00th=[  229], 80.00th=[  243], 90.00th=[  265], 95.00th=[  297],
     | 99.00th=[  502], 99.50th=[  660], 99.90th=[ 1467], 99.95th=[ 3163],
     | 99.99th=[41157]
  write: IOPS=24.5k, BW=95.6MiB/s (100MB/s)(32.3MiB/338msec); 0 zone resets
    slat (nsec): min=1883, max=40952k, avg=19011.89, stdev=779597.34
    clat (usec): min=56, max=40927, avg=240.38, stdev=786.06
     lat (usec): min=59, max=41599, avg=259.54, stdev=1111.91
    clat percentiles (usec):
     |  1.00th=[  119],  5.00th=[  161], 10.00th=[  174], 20.00th=[  188],
     | 30.00th=[  198], 40.00th=[  206], 50.00th=[  215], 60.00th=[  223],
     | 70.00th=[  233], 80.00th=[  245], 90.00th=[  265], 95.00th=[  306],
     | 99.00th=[  562], 99.50th=[  676], 99.90th=[ 1745], 99.95th=[ 3228],
     | 99.99th=[41157]
  lat (usec)   : 100=0.71%, 250=83.42%, 500=14.71%, 750=0.82%, 1000=0.16%
  lat (msec)   : 2=0.10%, 4=0.05%, 10=0.01%, 50=0.02%
  cpu          : usr=0.45%, sys=1.29%, ctx=16835, majf=1, minf=17
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=8115,8269,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=93.8MiB/s (98.3MB/s), 93.8MiB/s-93.8MiB/s (98.3MB/s-98.3MB/s), io=31.7MiB (33.2MB), run=338-338msec
  WRITE: bw=95.6MiB/s (100MB/s), 95.6MiB/s-95.6MiB/s (100MB/s-100MB/s), io=32.3MiB (33.9MB), run=338-338msec

Disk stats (read/write):
  sdb: ios=4919/4898, merge=0/0, ticks=1092/1100, in_queue=2192, util=62.10%

#  fio --rw=randrw --bs=4k --runtime=1m --direct=1 --filename=/dev/sdb --name=job1 --ioengine=libaio --numjobs=16 --thread --group_reporting
job1: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1
...
fio-3.27
Starting 16 threads

job1: (groupid=0, jobs=16): err= 0: pid=2200: Mon Jun  6 15:07:55 2022
  read: IOPS=21.6k, BW=84.5MiB/s (88.6MB/s)(31.7MiB/375msec)
    slat (nsec): min=1819, max=41118k, avg=35205.76, stdev=1116543.78
    clat (usec): min=29, max=41041, avg=282.38, stdev=911.00
     lat (usec): min=58, max=41810, avg=317.77, stdev=1447.16
    clat percentiles (usec):
     |  1.00th=[  106],  5.00th=[  165], 10.00th=[  184], 20.00th=[  204],
     | 30.00th=[  223], 40.00th=[  239], 50.00th=[  253], 60.00th=[  265],
     | 70.00th=[  277], 80.00th=[  297], 90.00th=[  326], 95.00th=[  388],
     | 99.00th=[  619], 99.50th=[  725], 99.90th=[ 2114], 99.95th=[ 2278],
     | 99.99th=[41157]
  write: IOPS=22.1k, BW=86.1MiB/s (90.3MB/s)(32.3MiB/375msec); 0 zone resets
    slat (nsec): min=1947, max=41105k, avg=30342.79, stdev=1009523.52
    clat (usec): min=2, max=2263, avg=267.68, stdev=119.79
     lat (usec): min=58, max=41773, avg=298.21, stdev=1025.24
    clat percentiles (usec):
     |  1.00th=[  109],  5.00th=[  169], 10.00th=[  186], 20.00th=[  206],
     | 30.00th=[  225], 40.00th=[  243], 50.00th=[  255], 60.00th=[  269],
     | 70.00th=[  281], 80.00th=[  297], 90.00th=[  334], 95.00th=[  420],
     | 99.00th=[  635], 99.50th=[  775], 99.90th=[ 2114], 99.95th=[ 2212],
     | 99.99th=[ 2278]
  lat (usec)   : 4=0.01%, 50=0.01%, 100=0.71%, 250=45.87%, 500=50.12%
  lat (usec)   : 750=2.78%, 1000=0.23%
  lat (msec)   : 2=0.15%, 4=0.10%, 50=0.02%
  cpu          : usr=0.51%, sys=1.54%, ctx=16625, majf=0, minf=16
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=8115,8269,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
   READ: bw=84.5MiB/s (88.6MB/s), 84.5MiB/s-84.5MiB/s (88.6MB/s-88.6MB/s), io=31.7MiB (33.2MB), run=375-375msec
  WRITE: bw=86.1MiB/s (90.3MB/s), 86.1MiB/s-86.1MiB/s (90.3MB/s-90.3MB/s), io=32.3MiB (33.9MB), run=375-375msec

Disk stats (read/write):
  sdb: ios=3859/3837, merge=0/0, ticks=961/971, in_queue=1931, util=60.17%

(In reply to Tingting Mao from comment #25)
> Hi Eric,
> 
> I have tried to verify this bug to compare between qemu-kvm-7.0.0-4.el9 and
> qemu-kvm-7.0.0-5.el9. The results indicate that:
>  - For qemu-nbd --list, the result is the same between qemu-kvm-7.0.0-4.el9
> and qemu-kvm-7.0.0-5.el9

Not quite.  Notice below...

>  - For performance, it indeed improved.
>    - For read, IOPS=23.0k VS IOPS=24.0k
>    - For write, IOPS=23.4k VS IOPS=24.5k
>    - For randread, IOPS=20.1k VS IOPS=21.6k
>    - For randwrite, IOPS=20.5k VS IOPS=22.1k

Good to hear!

> 
> Could you please help to check whether the result is okay? And need I to
> cover any other scenarios for this bug verification?
> 

> Results:
> =============================================================================
> ===========================================================
> In qemu-kvm-7.0.0-4.el9:
> # qemu-nbd --list 
> exports available: 1
>  export: ''
>   size:  4194304
>   flags: 0xced ( flush fua trim zeroes df cache fast-zero )

> ===================================================
> In qemu-kvm-7.0.0-5.el9:
> # qemu-nbd --list
> exports available: 1
>  export: ''
>   size:  4194304
>   flags: 0xded ( flush fua trim zeroes df multi cache fast-zero )

The addition of the 'multi' flag is what we wanted to see!

The improved performance is also nice.  But you have indeed verified that the setting is advertised correctly.

Thanks Eric, set this bug as verified accordingly.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: qemu-kvm security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:7967