QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.

Failed Test on 

Red Hat Enterprise Linux release 9.3 Beta (Plow)
5.14.0-332.el9.x86_64
qemu-kvm-8.0.0-6.el9.x86_64
seabios-bin-1.16.1-1.el9.noarch
edk2-ovmf-20230301gitf80f052277c8-5.el9.noarch
virtio-win-prewhql-0.1-238.iso

steps 
1. create vhost vdpa device

  modprobe -r vhost-vdpa
  modprobe -r vdpa-sim-blk

  modprobe vhost-vdpa
  modprobe vdpa-sim-blk

  vdpa dev add mgmtdev vdpasim_blk name blk0
  vdpa dev add mgmtdev vdpasim_blk name blk1
  vdpa dev list -jp
  ls /dev/vhost-vdpa*
  [ $? -ne 0 ] && echo "wrong create vdpa device"


2. boot vm with driver





/usr/libexec/qemu-kvm \
  -name testvm \
  -machine q35 \
  -m  6G \
  -smp 2 \
  -cpu host,+kvm_pv_unhalt \
  -device ich9-usb-ehci1,id=usb1 \
  -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
   \
   \
  -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x3,chassis=1 \
  -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x3.0x1,bus=pcie.0,chassis=2 \
  -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x3.0x2,bus=pcie.0,chassis=3 \
  -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x3.0x3,bus=pcie.0,chassis=4 \
  -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x3.0x4,bus=pcie.0,chassis=5 \
  -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x3.0x5,bus=pcie.0,chassis=6 \
  -device pcie-root-port,id=pcie-root-port-6,port=0x6,addr=0x3.0x6,bus=pcie.0,chassis=7 \
  -device pcie-root-port,id=pcie-root-port-7,port=0x7,addr=0x3.0x7,bus=pcie.0,chassis=8 \
  -device pcie-root-port,id=pcie_extra_root_port_0,bus=pcie.0,addr=0x4  \
  -object iothread,id=iothread0 \
  -device virtio-scsi-pci,id=scsi0,bus=pcie-root-port-0 \
  -blockdev driver=qcow2,file.driver=file,cache.direct=off,cache.no-flush=on,file.filename=/home/kvm_autotest_root/images/rhel930-64-virtio-scsi.qcow2,node-name=drive_image1,file.aio=threads   \
  -device scsi-hd,id=os,drive=drive_image1,bus=scsi0.0,bootindex=0,serial=OS_DISK   \
  \
  -blockdev node-name=file_stg1,driver=virtio-blk-vhost-vdpa,path=/dev/vhost-vdpa-0,cache.direct=on,cache.no-flush=off,discard=unmap \
  -blockdev node-name=drive_stg1,driver=raw,cache.direct=on,cache.no-flush=off,file=file_stg1 \
  -device virtio-blk-pci,iothread=iothread0,serial=stg1,bus=pcie.0-root-port-4,addr=0x0,write-cache=on,id=stg1,drive=drive_stg1,rerror=report,werror=report \
  \
  -blockdev node-name=file_stg2,driver=virtio-blk-vhost-vdpa,path=/dev/vhost-vdpa-1,cache.direct=on,cache.no-flush=off,discard=unmap \
  -blockdev node-name=drive_stg2,driver=raw,cache.direct=on,cache.no-flush=off,file=file_stg2 \
  -device scsi-hd,id=stg2,drive=drive_stg2,serial=stg2 \
  -vnc :5 \
  -monitor stdio \
  -qmp tcp:0:5955,server=on,wait=off \
  -device virtio-net-pci,mac=9a:b5:b6:b1:b2:b7,id=nic1,netdev=nicpci,bus=pcie-root-port-7 \
  -netdev tap,id=nicpci \
  -boot menu=on,reboot-timeout=1000,strict=off \
  \

It is failed 
qemu-kvm: -blockdev node-name=file_stg1,driver=virtio-blk-vhost-vdpa,path=/dev/vhost-vdpa-0,cache.direct=on,cache.no-flush=off,discard=unmap: Unknown driver 'virtio-blk-vhost-vdpa'

(In reply to Stefano Garzarella from comment #10)
> (In reply to qing.wang from comment #7)
> > Failed Test on 
> > 
> > Red Hat Enterprise Linux release 9.3 Beta (Plow)
> > 5.14.0-332.el9.x86_64
> > qemu-kvm-8.0.0-6.el9.x86_64
> > seabios-bin-1.16.1-1.el9.noarch
> > edk2-ovmf-20230301gitf80f052277c8-5.el9.noarch
> > virtio-win-prewhql-0.1-238.iso
> 
> 
> @qinwang maybe you need to install also
> qemu-kvm-block-blkio-8.0.0-6.el9.x86_64
> 
> IIUC blkio drivers are built in a QEMU module

it already is installed.

root@dell-per440-07 /home/kar/tp-qemu/qemu/tests $ rpm -qa|grep qemu-kvm-block-blkio
qemu-kvm-block-blkio-8.0.0-6.el9.x86_64
qemu-kvm-block-blkio-debuginfo-8.0.0-6.el9.x86_64

root@dell-per440-07 /home/kar/tp-qemu/qemu/tests $ rpm -qa|grep libblkio
libblkio-1.3.0-1.el9.x86_64


Hi

I think the problem is in block/meson.build and scripts/modules/module_block.py.

scripts/modules/module_block.py is supposed to fill `block_driver_modules[]` array, but we need to update it to handle libblkio drivers in the right way.

Current block_driver_modules[] doesn't include them:

(gdb) p block_driver_modules
$30 = {{format_name = 0x55555605c69d <str.67.llvm> "http", 
    protocol_name = 0x55555605c69d <str.67.llvm> "http", 
    library_name = 0x55555608c68f <str.155.llvm> "curl"}, {
    format_name = 0x55555605c6a6 <str.68.llvm> "https", 
    protocol_name = 0x55555605c6a6 <str.68.llvm> "https", 
    library_name = 0x55555608c68f <str.155.llvm> "curl"}, {
    format_name = 0x55555605dbd9 <str.62.llvm> "ftp", protocol_name = 0x55555605dbd9 <str.62.llvm> "ftp", 
    library_name = 0x55555608c68f <str.155.llvm> "curl"}, {
    format_name = 0x55555605c671 <str.63.llvm> "ftps", 
    protocol_name = 0x55555605c671 <str.63.llvm> "ftps", 
    library_name = 0x55555608c68f <str.155.llvm> "curl"}, {
    format_name = 0x55555608c69a <str.3.llvm> "rbd", protocol_name = 0x55555608c69a <str.3.llvm> "rbd", 
    library_name = 0x55555608c69a <str.3.llvm> "rbd"}}

We also need to fix a typo in .distro/qemu-kvm.spec: s/virtio-blk-vdpa-blk/virtio-blk-vhost-vdpa

Thanks, Stefano. I will send patches.

Module loading fix posted upstream:
https://lore.kernel.org/qemu-devel/20230703103524.162855-1-stefanha@redhat.com/T/#u

I will fix the qemu-kvm.spec typo in the backport.

QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.

VM can not boot, it hangs on "Probing EDD (edd=off to disable)..."

Red Hat Enterprise Linux release 9.3 Beta (Plow)
5.14.0-332.el9.x86_64
qemu-kvm-8.0.0-8.el9.x86_64
seabios-bin-1.16.1-1.el9.noarch
edk2-ovmf-20230524-2.el9.noarch

/usr/libexec/qemu-kvm \
  -name testvm \
  -machine q35 \
  -m  6G \
  -smp 2 \
  -cpu host,+kvm_pv_unhalt \
  -device ich9-usb-ehci1,id=usb1 \
  -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
   \
   \
  -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x3,chassis=1 \
  -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x3.0x1,bus=pcie.0,chassis=2 \
  -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x3.0x2,bus=pcie.0,chassis=3 \
  -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x3.0x3,bus=pcie.0,chassis=4 \
  -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x3.0x4,bus=pcie.0,chassis=5 \
  -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x3.0x5,bus=pcie.0,chassis=6 \
  -device pcie-root-port,id=pcie-root-port-6,port=0x6,addr=0x3.0x6,bus=pcie.0,chassis=7 \
  -device pcie-root-port,id=pcie-root-port-7,port=0x7,addr=0x3.0x7,bus=pcie.0,chassis=8 \
  -device pcie-root-port,id=pcie_extra_root_port_0,bus=pcie.0,addr=0x4  \
  -object iothread,id=iothread0 \
  -device virtio-scsi-pci,id=scsi0,bus=pcie-root-port-0 \
  -blockdev driver=qcow2,file.driver=file,cache.direct=off,cache.no-flush=on,file.filename=/home/kvm_autotest_root/images/rhel930-64-virtio-scsi.qcow2,node-name=drive_image1,file.aio=threads   \
  -device scsi-hd,id=os,drive=drive_image1,bus=scsi0.0,bootindex=0,serial=OS_DISK   \
  \
 -blockdev node-name=prot_stg1,driver=virtio-blk-vhost-vdpa,path=/dev/vhost-vdpa-0,cache.direct=on \
  -blockdev node-name=fmt_stg1,driver=raw,file=prot_stg1 \
  -device virtio-blk-pci,bus=pcie-root-port-4,addr=0,id=stg1,drive=fmt_stg1 \
  -vnc :5 \
  -monitor stdio \
  -qmp tcp:0:5955,server=on,wait=off \
  -device virtio-net-pci,mac=9a:b5:b6:b1:b2:b7,id=nic1,netdev=nicpci,bus=pcie-root-port-7 \
  -netdev tap,id=nicpci \
  -boot menu=on,reboot-timeout=1000,strict=off \
  \

Do you know what is wrong with my command line ?

Can you interrupt grub boot, and then edit the boot command line.

- Remove anything that says "rhgb quiet"

- Add "debug"

That might give more information about where it's really hanging.

If possible it'd be nice if you could attach a virtual serial console, as
that will help to capture the full boot messages.

(In reply to Richard W.M. Jones from comment #24)
> Can you interrupt grub boot, and then edit the boot command line.
> 
> - Remove anything that says "rhgb quiet"
> 
> - Add "debug"
> 
> That might give more information about where it's really hanging.
> 
> If possible it'd be nice if you could attach a virtual serial console, as
> that will help to capture the full boot messages.


It hangs on the very beginning stage, no debug info display

http://fileshare.hosts.qa.psi.pek2.redhat.com/pub/section2/images_backup/qbugs/2213317/2023-07-19/boot.png

I understand that.  However the EDD message is typically not where the hang actually
happens in my experience.

I would strongly suggest attaching a serial console, then editing the Linux command
line (with grub) to remove "rhgb quiet", and add "debug console=/dev/ttyS0", then see
what is printed on the serial console.

(In reply to Richard W.M. Jones from comment #26)
> I understand that.  However the EDD message is typically not where the hang
> actually
> happens in my experience.
> 
> I would strongly suggest attaching a serial console, then editing the Linux
> command
> line (with grub) to remove "rhgb quiet", and add "debug console=/dev/ttyS0",
> then see
> what is printed on the serial console.

the console is added in kernel parameter, 
http://fileshare.hosts.qa.psi.pek2.redhat.com/pub/section2/images_backup/qbugs/2213317/2023-07-20/grub.png

No log in the serial log 

/usr/libexec/qemu-kvm \
  -name testvm \
  -machine q35 \
  -m  6G \
  -smp 2 \
  -cpu host,+kvm_pv_unhalt \
  -device ich9-usb-ehci1,id=usb1 \
  -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
   \
   \
  -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x3,chassis=1 \
  -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x3.0x1,bus=pcie.0,chassis=2 \
  -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x3.0x2,bus=pcie.0,chassis=3 \
  -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x3.0x3,bus=pcie.0,chassis=4 \
  -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x3.0x4,bus=pcie.0,chassis=5 \
  -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x3.0x5,bus=pcie.0,chassis=6 \
  -device pcie-root-port,id=pcie-root-port-6,port=0x6,addr=0x3.0x6,bus=pcie.0,chassis=7 \
  -device pcie-root-port,id=pcie-root-port-7,port=0x7,addr=0x3.0x7,bus=pcie.0,chassis=8 \
  -device pcie-root-port,id=pcie_extra_root_port_0,bus=pcie.0,addr=0x4  \
  -object iothread,id=iothread0 \
  -device virtio-scsi-pci,id=scsi0,bus=pcie-root-port-0 \
  -blockdev driver=qcow2,file.driver=file,cache.direct=off,cache.no-flush=on,file.filename=/home/kvm_autotest_root/images/rhel930-64-virtio-scsi.qcow2,node-name=drive_image1,file.aio=threads   \
  -device scsi-hd,id=os,drive=drive_image1,bus=scsi0.0,bootindex=0,serial=OS_DISK   \
  \
 -blockdev node-name=prot_stg1,driver=virtio-blk-vhost-vdpa,path=/dev/vhost-vdpa-0,cache.direct=on \
  -blockdev node-name=fmt_stg1,driver=raw,file=prot_stg1 \
  -device virtio-blk-pci,bus=pcie-root-port-4,addr=0,id=stg1,drive=fmt_stg1,bootindex=2 \
  -vnc :5 \
  -monitor stdio \
  -qmp tcp:0:5955,server=on,wait=off \
  -device virtio-net-pci,mac=9a:b5:b6:b1:b2:b7,id=nic1,netdev=nicpci,bus=pcie-root-port-7 \
  -netdev tap,id=nicpci \
  -boot menu=on,reboot-timeout=1000,strict=off \
  \
  -chardev socket,id=socket-serial,path=/var/tmp/socket-serial,logfile=/var/tmp/file-serial.log,mux=on,server=on,wait=off \
  -serial chardev:socket-serial \
  -chardev file,path=/var/tmp/file-bios.log,id=file-bios \
  -device isa-debugcon,chardev=file-bios,iobase=0x402 \





root@dell-per6515-03 /home $ cat  -A /var/tmp/file-serial.log 
^[[H^[[J^[[1;1H^[[?25l^[[m^[[H^[[J^[[1;1H^[[2;32HGRUB version 2.06$
^M$
^M^[[m^[[4;2H+----------------------------------------------------------------------------+^[[5;2H|^[[5;79H|^[[6;2H|^[[6;79H|^[[7;2H|^[[7;79H|^[[8;2H|^[[8;79H|^[[9;2H|^[[9;79H|^[[10;2H|^[[10;79H|^[[11;2H|^[[11;79H|^[[12;2H|^[[12;79H|^[[13;2H|^[[13;79H|^[[14;2H|^[[14;79H|^[[15;2H|^[[15;79H|^[[16;2H|^[[16;79H|^[[17;2H+----------------------------------------------------------------------------+^[[m^[[18;2H^[[19;2H^[[m     Use the ^ and v keys to select which entry is highlighted.          $
^M      Press enter to boot the selected OS, `e' to edit the commands       $
^M      before booting or `c' for a command-line.                           ^[[5;80H ^[[7m^[[5;3H*Red Hat Enterprise Linux (5.14.0-332.el9.x86_64) 9.3 (Plow)                ^[[m^[[5;78H^[[m^[[m^[[6;3H Red Hat Enterprise Linux (0-rescue-df1ed2684a0341ee8eb28b5f64e0bc73) 9.3 (>^[[m^[[6;78H^[[m^[[m^[[7;3H                                                                            ^[[m^[[7;78H^[[m^[[m^[[8;3H                                                                            ^[[m^[[8;78H^[[m^[[m^[[9;3H                                                                            ^[[m^[[9;78H^[[m^[[m^[[10;3H                                                                            ^[[m^[[10;78H^[[m^[[m^[[11;3H                                                                            ^[[m^[[11;78H^[[m^[[m^[[12;3H                                                                            ^[[m^[[12;78H^[[m^[[m^[[13;3H                                                                            ^[[m^[[13;78H^[[m^[[m^[[14;3H                                                                            ^[[m^[[14;78H^[[m^[[m^[[15;3H                                                                            ^[[m^[[15;78H^[[m^[[m^[[16;3H                                                                            ^[[m^[[16;78H^[[m^[[16;80H ^[[5;78H^[[22;1H   The highlighted entry will be executed automatically in 5s.                 ^[[5;78H^[[22;1H   The highlighted entry will be executed automatically in 4s.                 ^[[5;78H^[[22;1H   The highlighted entry will be executed automatically in 3s.                 ^[[5;78H^[[22;1H   The highlighted entry will be executed automatically in 2s.                 ^[[5;78H^[[22;1H   The highlighted entry will be executed automatically in 1s.                 ^[[5;78H^[[22;1H   The highlighted entry will be executed automatically in 0s.                 ^[[5;78H^[[?25h^[[H^[[J^[[1;1H^[[H^[[J^[[1;1H  Booting `Red Hat Enterprise Linux (5.14.0-332.el9.x86_64) 9.3 (Plow)'$
^M$

The kernel printed nothing at all on the serial port, not even the EDD
message, so there's obviously something wrong with your serial port configuration.
I'd try to fix that first so you can get the all important kernel messages out,
otherwise this bug is going to be impossible to debug.

Qing shared the access with the machine and I think the problem is that vhost-vdpa devices still require pinning all the guest pages.
So using 4k pages with a 6G guest ram, can take a lot of time.

This requirement may disappear if we will backport commit 4bb94d2de2fa ("vdpa_sim: add support for user VA") [1] for the vDPA simulators but in general HW device don't support this yet, so for now my advice is to use hugepages with vhost-vdpa.

Something like this:

  # allocate 3100 hugepages (2M): 6G/2M = 3072 + some extra pages
  echo 3100  > /proc/sys/vm/nr_hugepages

  /usr/libexec/qemu-kvm \
  ...
  -machine q35,memory-backend=mem \
  -object memory-backend-file,share=on,id=mem,size=6G,mem-path=/dev/hugepages/libvirt/qemu \

Or as Qing suggested a memory-backend-memfd since Linux should support transparent huge-pages of shmem/memfd memory:

  /usr/libexec/qemu-kvm \
  ...
  -machine q35,memory-backend=mem \
  -object memory-backend-memfd,id=mem,size=6G,share=on \


We also need to increase memlock limits in /etc/security/limits.conf:

* soft memlock unlimited
* hard memlock unlimited


[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4bb94d2de2fa90aa8d4e1ce940b1b2f7708cf141

VM can not boot with 

qemu-storage-daemon \
--chardev socket,path=/home/qsd/qmp.sock,server=on,wait=off,id=char1 \
--monitor chardev=char1 \
--blockdev file,filename=/home/kvm_autotest_root/images/qsd1.qcow2,node-name=prot1 \
--blockdev qcow2,file=prot1,node-name=disk1 \
--export vhost-user-blk,id=exp1,addr.type=unix,addr.path=/home/qsd/vhost1.sock,node-name=disk1,writable=on \
--blockdev file,filename=/home/kvm_autotest_root/images/qsd2.qcow2,node-name=prot2 \
--blockdev qcow2,file=prot2,node-name=disk2 \
--export vhost-user-blk,id=exp2,addr.type=unix,addr.path=/home/qsd/vhost2.sock,node-name=disk2,writable=on \


2.Boot VM with virtio-blk-vhost-userdriver

/usr/libexec/qemu-kvm \
  -name testvm \
  -machine q35,memory-backend=mem \
  -object memory-backend-memfd,id=mem,size=6G,share=on \
  -m  6G \
  -smp 2 \
  -cpu host,+kvm_pv_unhalt \
  -device ich9-usb-ehci1,id=usb1 \
  -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
   \
   \
  -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x3,chassis=1 \
  -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x3.0x1,bus=pcie.0,chassis=2 \
  -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x3.0x2,bus=pcie.0,chassis=3 \
  -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x3.0x3,bus=pcie.0,chassis=4 \
  -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x3.0x4,bus=pcie.0,chassis=5 \
  -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x3.0x5,bus=pcie.0,chassis=6 \
  -device pcie-root-port,id=pcie-root-port-6,port=0x6,addr=0x3.0x6,bus=pcie.0,chassis=7 \
  -device pcie-root-port,id=pcie-root-port-7,port=0x7,addr=0x3.0x7,bus=pcie.0,chassis=8 \
  -device pcie-root-port,id=pcie_extra_root_port_0,bus=pcie.0,addr=0x4  \
  -object iothread,id=iothread0 \
  -device virtio-scsi-pci,id=scsi0,bus=pcie-root-port-0,iothread=iothread0 \
  -blockdev driver=qcow2,file.driver=file,cache.direct=off,cache.no-flush=on,file.filename=/home/kvm_autotest_root/images/rhel930-64-virtio-scsi.qcow2,node-name=drive_image1,file.aio=threads   \
  -device scsi-hd,id=os,drive=drive_image1,bus=scsi0.0,bootindex=0,serial=OS_DISK   \
  \
  -blockdev node-name=prot_stg0,driver=virtio-blk-vhost-user,path=/home/qsd/vhost1.sock,cache.direct=on \
  -blockdev node-name=fmt_stg0,driver=raw,file=prot_stg0 \
  -device virtio-blk-pci,iothread=iothread0,bus=pcie-root-port-4,addr=0,id=stg0,drive=fmt_stg0,bootindex=1 \
  \
  -blockdev node-name=prot_stg1,driver=virtio-blk-vhost-user,path=/home/qsd/vhost2.sock,cache.direct=on \
  -blockdev node-name=fmt_stg1,driver=raw,file=prot_stg1 \
  -device scsi-hd,id=stg1,drive=fmt_stg1,bootindex=2 \
  -vnc :5 \
  -monitor stdio \
  -qmp tcp:0:5955,server=on,wait=off \
  -device virtio-net-pci,mac=9a:b5:b6:b1:b2:b7,id=nic1,netdev=nicpci,bus=pcie-root-port-7 \
  -netdev tap,id=nicpci \
  -boot menu=on,reboot-timeout=1000,strict=off \
  \
  -chardev socket,id=socket-serial,path=/var/tmp/socket-serial,logfile=/var/tmp/file-serial.log,mux=on,server=on,wait=off \
  -serial chardev:socket-serial \
  -chardev file,path=/var/tmp/file-bios.log,id=file-bios \
  -device isa-debugcon,chardev=file-bios,iobase=0x402 \
  \
  -chardev socket,id=socket-qmp,path=/var/tmp/socket-qmp,logfile=/var/tmp/file-qmp.log,mux=on,server=on,wait=off \
  -mon chardev=socket-qmp,mode=control \
  -chardev socket,id=socket-hmp,path=/var/tmp/socket-hmp,logfile=/var/tmp/file-hmp.log,mux=on,server=on,wait=off \
  -mon chardev=socket-hmp,mode=readline \

Boot failed :
qemu-kvm: -blockdev node-name=prot_stg0,driver=virtio-blk-vhost-user,path=/home/qsd/vhost1.sock,cache.direct=on: Could not open '/home/qsd/vhost1.sock': No such device or address

The driver virtio-blk-vhost-vdpa works on

Red Hat Enterprise Linux release 9.3 Beta (Plow)
5.14.0-340.el9.x86_64
qemu-kvm-8.0.0-8.el9.x86_64
seabios-bin-1.16.1-1.el9.noarch
edk2-ovmf-20230524-2.el9.noarch
libvirt-9.3.0-2.el9.x86_64


Test steps:
1. parare vhost vdpa disks on host

  modprobe vhost-vdpa
  modprobe vdpa-sim-blk
  vdpa dev add mgmtdev vdpasim_blk name blk0
  vdpa dev add mgmtdev vdpasim_blk name blk1
  vdpa dev list -jp
  ls /dev/vhost-vdpa*
  [ $? -ne 0 ] && echo "wrong create vdpa device"

2. boot VM with virtio-blk-vhost-vdpa driver
/usr/libexec/qemu-kvm \
  -name testvm \
  -machine q35,memory-backend=mem \
  -object memory-backend-memfd,id=mem,size=6G,share=on \
  -m  6G \
  -smp 2 \
  -cpu host,+kvm_pv_unhalt \
  -device ich9-usb-ehci1,id=usb1 \
  -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
   \
   \
  -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x3,chassis=1 \
  -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x3.0x1,bus=pcie.0,chassis=2 \
  -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x3.0x2,bus=pcie.0,chassis=3 \
  -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x3.0x3,bus=pcie.0,chassis=4 \
  -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x3.0x4,bus=pcie.0,chassis=5 \
  -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x3.0x5,bus=pcie.0,chassis=6 \
  -device pcie-root-port,id=pcie-root-port-6,port=0x6,addr=0x3.0x6,bus=pcie.0,chassis=7 \
  -device pcie-root-port,id=pcie-root-port-7,port=0x7,addr=0x3.0x7,bus=pcie.0,chassis=8 \
  -device pcie-root-port,id=pcie_extra_root_port_0,bus=pcie.0,addr=0x4  \
  -object iothread,id=iothread0 \
  -device virtio-scsi-pci,id=scsi0,bus=pcie-root-port-0,iothread=iothread0 \
  -blockdev driver=qcow2,file.driver=file,cache.direct=off,cache.no-flush=on,file.filename=/home/kvm_autotest_root/images/rhel930-64-virtio-scsi.qcow2,node-name=drive_image1,file.aio=threads   \
  -device scsi-hd,id=os,drive=drive_image1,bus=scsi0.0,bootindex=0,serial=OS_DISK   \
  \
  -blockdev node-name=prot_stg0,driver=virtio-blk-vhost-vdpa,path=/dev/vhost-vdpa-0,cache.direct=on \
  -blockdev node-name=fmt_stg0,driver=raw,file=prot_stg0 \
  -device virtio-blk-pci,iothread=iothread0,bus=pcie-root-port-4,addr=0,id=stg0,drive=fmt_stg0,bootindex=1 \
  \
  -blockdev node-name=prot_stg1,driver=virtio-blk-vhost-vdpa,path=/dev/vhost-vdpa-1,cache.direct=on \
  -blockdev node-name=fmt_stg1,driver=raw,file=prot_stg1 \
  -device scsi-hd,id=stg1,drive=fmt_stg1,bootindex=2 \
  -vnc :5 \
  -monitor stdio \
  -qmp tcp:0:5955,server=on,wait=off \
  -device virtio-net-pci,mac=9a:b5:b6:b1:b2:b7,id=nic1,netdev=nicpci,bus=pcie-root-port-7 \
  -netdev tap,id=nicpci \
  -boot menu=on,reboot-timeout=1000,strict=off \
  \
  -chardev socket,id=socket-serial,path=/var/tmp/socket-serial,logfile=/var/tmp/file-serial.log,mux=on,server=on,wait=off \
  -serial chardev:socket-serial \
  -chardev file,path=/var/tmp/file-bios.log,id=file-bios \
  -device isa-debugcon,chardev=file-bios,iobase=0x402 \
  \
  -chardev socket,id=socket-qmp,path=/var/tmp/socket-qmp,logfile=/var/tmp/file-qmp.log,mux=on,server=on,wait=off \
  -mon chardev=socket-qmp,mode=control \
  -chardev socket,id=socket-hmp,path=/var/tmp/socket-hmp,logfile=/var/tmp/file-hmp.log,mux=on,server=on,wait=off \
  -mon chardev=socket-hmp,mode=readline \

3. login guest and execute simple io
cat guest_io.sh 

function tests_failed() {
        exit_code="$?"
        echo "Test failed: $1"
        exit "${exit_code}"
}
vdpa_devs=`lsblk -nd|grep 128M|awk '{print $1}'`
echo ${vdpa_devs}

for dev in ${vdpa_devs};do
  echo "$dev"
  mkfs.xfs -f /dev/${dev} || tests_failed "format"
  mkdir -p /home/${dev}
  mount /dev/${dev} /home/${dev} || tests_failed "mount"
  dd if=/dev/zero of=/home/${dev}/test.img count=100 bs=1M oflag=direct || tests_failed "IO"
  umount -fl /home/${dev}
done

4.unplug disks

{"execute": "device_del", "arguments": {"id": "stg1"}}

{"execute": "blockdev-del","arguments": {"node-name": "fmt_stg1"}}
{"execute": "blockdev-del","arguments": {"node-name":"prot_stg1"}}

{"execute": "device_del", "arguments": {"id": "stg0"}}

{"execute": "blockdev-del","arguments": {"node-name": "fmt_stg0"}}
{"execute": "blockdev-del","arguments": {"node-name":"prot_stg0"}}

5. check disks in guest, the disks should disappear

6. plug disks

{"execute": "blockdev-add", "arguments": {"node-name": "prot_stg0", "driver": "virtio-blk-vhost-vdpa",  "path": "/dev/vhost-vdpa-0","cache": {"direct": true, "no-flush": false}}}
{"execute": "blockdev-add", "arguments": {"node-name": "fmt_stg0", "driver": "raw",   "file": "prot_stg0"}}
{"execute": "device_add", "arguments": {"driver": "virtio-blk-pci", "id": "stg0", "drive": "fmt_stg0","bus":"pcie-root-port-4"}}

{"execute": "blockdev-add", "arguments": {"node-name": "prot_stg1", "driver": "virtio-blk-vhost-vdpa",  "path": "/dev/vhost-vdpa-1","cache": {"direct": true, "no-flush": false}}}
{"execute": "blockdev-add", "arguments": {"node-name": "fmt_stg1", "driver": "raw",   "file": "prot_stg1"}}
{"execute": "device_add", "arguments": {"driver": "scsi-hd", "id": "stg1", "drive": "fmt_stg1"}}

7.check disks in guest, the disks should exist

8. run io like step 3

(In reply to qing.wang from comment #30)
> VM can not boot with 

[...]

> Boot failed :
> qemu-kvm: -blockdev
> node-name=prot_stg0,driver=virtio-blk-vhost-user,path=/home/qsd/vhost1.sock,
> cache.direct=on: Could not open '/home/qsd/vhost1.sock': No such device or
> address

Argh, I replicated upstream. I think I introduced this issue while adding `fd` support for vhost-vdpa.

In QEMU we are using blkio_get_int("fd") to check if fd passing is supported or not, but in libblkio the `fd` property is shared with all the virtio-blk drivers and the error is returned only during blkio_connect().
So we can't use `blkio_get_int("fd")` in QEMU to check if the driver supports it or not.

@qinwang please open another BZ to track that issue, for now let's use vhost-vdpa or io_uring to test libblkio support in QEMU.

I'll try to fix that issue in QEMU or libblkio.

(In reply to Stefano Garzarella from comment #32)
> (In reply to qing.wang from comment #30)
> @qinwang please open another BZ to track that issue, for now
> let's use vhost-vdpa or io_uring to test libblkio support in QEMU.
> 
> I'll try to fix that issue in QEMU or libblkio.

Sounds good. We can make bz#2213317 dependent on the new BZ that addresses the virtio-blk-vhost-user "Could not open '/home/qsd/vhost1.sock': No such device or address" error.

(In reply to Stefan Hajnoczi from comment #33)
> (In reply to Stefano Garzarella from comment #32)
> > (In reply to qing.wang from comment #30)
> > @qinwang please open another BZ to track that issue, for now
> > let's use vhost-vdpa or io_uring to test libblkio support in QEMU.
> > 
> > I'll try to fix that issue in QEMU or libblkio.
> 
> Sounds good. We can make bz#2213317 dependent on the new BZ that addresses
> the virtio-blk-vhost-user "Could not open '/home/qsd/vhost1.sock': No such
> device or address" error.

Thanks, File a new bug Bug to track virtio-blk-vhost-user issue:
2225354 - [vdpa-blk] The new driver virtio-blk-vhost-user not work in VM booting

Pass test driver virtio-blk-vhost-user scenario:
multi disks booting, hotplug, options, installation test 
test driver virt-blk-vhost-vdpa scenario:
multi disks booting, hotplug, options, migration test

Red Hat Enterprise Linux release 9.3 Beta (Plow)
5.14.0-352.el9.x86_64
qemu-kvm-8.0.0-11.el9.x86_64
seabios-bin-1.16.1-1.el9.noarch
edk2-ovmf-20230524-2.el9.noarch
libvirt-9.5.0-5.el9.x86_64
virtio-win-prewhql-0.1-240.iso

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: qemu-kvm security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6368