Bug 2225354

Summary: [vdpa-blk] The new driver virtio-blk-vhost-user not work in VM booting
Product: Red Hat Enterprise Linux 9 Reporter: qing.wang <qinwang>
Component: qemu-kvmAssignee: Stefano Garzarella <sgarzare>
qemu-kvm sub component: virtio-blk,scsi QA Contact: qing.wang <qinwang>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: aliang, chayang, coli, jinzhao, juzhang, kwolf, lijin, qizhu, sgarzare, stefanha, virt-maint, xuwei, zhenyzha
Version: 9.3Keywords: Triaged
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: qemu-kvm-8.0.0-11.el9 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-11-07 08:28:05 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2213317    

Description qing.wang 2023-07-25 04:06:08 UTC
Description of problem:
Boot VM , the disks with new driver virtio-blk-vhost-user. 
It is failed :
Could not open '/home/qsd/vhost1.sock': No such device or address

Version-Release number of selected component (if applicable):
Red Hat Enterprise Linux release 9.3 Beta (Plow)
5.14.0-340.el9.x86_64
qemu-kvm-8.0.0-8.el9.x86_64
seabios-bin-1.16.1-1.el9.noarch
edk2-ovmf-20230524-2.el9.noarch
libvirt-9.3.0-2.el9.x86_64
libblkio-1.3.0-1.el9.x86_64

How reproducible:
100%

Steps to Reproduce:
1.Start QSD daemon

qemu-storage-daemon \
--chardev socket,path=/home/qsd/qmp.sock,server=on,wait=off,id=char1 \
--monitor chardev=char1 \
--blockdev file,filename=/home/kvm_autotest_root/images/qsd1.qcow2,node-name=prot1 \
--blockdev qcow2,file=prot1,node-name=disk1 \
--export vhost-user-blk,id=exp1,addr.type=unix,addr.path=/home/qsd/vhost1.sock,node-name=disk1,writable=on \
--blockdev file,filename=/home/kvm_autotest_root/images/qsd2.qcow2,node-name=prot2 \
--blockdev qcow2,file=prot2,node-name=disk2 \
--export vhost-user-blk,id=exp2,addr.type=unix,addr.path=/home/qsd/vhost2.sock,node-name=disk2,writable=on \


2.Boot VM with virtio-blk-vhost-user driver

/usr/libexec/qemu-kvm \
  -name testvm \
  -machine q35,memory-backend=mem \
  -object memory-backend-memfd,id=mem,size=6G,share=on \
  -m  6G \
  -smp 2 \
  -cpu host,+kvm_pv_unhalt \
  -device ich9-usb-ehci1,id=usb1 \
  -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
   \
   \
  -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x3,chassis=1 \
  -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x3.0x1,bus=pcie.0,chassis=2 \
  -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x3.0x2,bus=pcie.0,chassis=3 \
  -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x3.0x3,bus=pcie.0,chassis=4 \
  -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x3.0x4,bus=pcie.0,chassis=5 \
  -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x3.0x5,bus=pcie.0,chassis=6 \
  -device pcie-root-port,id=pcie-root-port-6,port=0x6,addr=0x3.0x6,bus=pcie.0,chassis=7 \
  -device pcie-root-port,id=pcie-root-port-7,port=0x7,addr=0x3.0x7,bus=pcie.0,chassis=8 \
  -device pcie-root-port,id=pcie_extra_root_port_0,bus=pcie.0,addr=0x4  \
  -object iothread,id=iothread0 \
  -device virtio-scsi-pci,id=scsi0,bus=pcie-root-port-0,iothread=iothread0 \
  -blockdev driver=qcow2,file.driver=file,cache.direct=off,cache.no-flush=on,file.filename=/home/kvm_autotest_root/images/rhel930-64-virtio-scsi.qcow2,node-name=drive_image1,file.aio=threads   \
  -device scsi-hd,id=os,drive=drive_image1,bus=scsi0.0,bootindex=0,serial=OS_DISK   \
  \
  -blockdev node-name=prot_stg0,driver=virtio-blk-vhost-user,path=/home/qsd/vhost1.sock,cache.direct=on \
  -blockdev node-name=fmt_stg0,driver=raw,file=prot_stg0 \
  -device virtio-blk-pci,iothread=iothread0,bus=pcie-root-port-4,addr=0,id=stg0,drive=fmt_stg0,bootindex=1 \
  \
  -blockdev node-name=prot_stg1,driver=virtio-blk-vhost-user,path=/home/qsd/vhost2.sock,cache.direct=on \
  -blockdev node-name=fmt_stg1,driver=raw,file=prot_stg1 \
  -device scsi-hd,id=stg1,drive=fmt_stg1,bootindex=2 \
  -vnc :5 \
  -monitor stdio \
  -qmp tcp:0:5955,server=on,wait=off \
  -device virtio-net-pci,mac=9a:b5:b6:b1:b2:b7,id=nic1,netdev=nicpci,bus=pcie-root-port-7 \
  -netdev tap,id=nicpci \
  -boot menu=on,reboot-timeout=1000,strict=off \
  \
  -chardev socket,id=socket-serial,path=/var/tmp/socket-serial,logfile=/var/tmp/file-serial.log,mux=on,server=on,wait=off \
  -serial chardev:socket-serial \
  -chardev file,path=/var/tmp/file-bios.log,id=file-bios \
  -device isa-debugcon,chardev=file-bios,iobase=0x402 \
  \
  -chardev socket,id=socket-qmp,path=/var/tmp/socket-qmp,logfile=/var/tmp/file-qmp.log,mux=on,server=on,wait=off \
  -mon chardev=socket-qmp,mode=control \
  -chardev socket,id=socket-hmp,path=/var/tmp/socket-hmp,logfile=/var/tmp/file-hmp.log,mux=on,server=on,wait=off \
  -mon chardev=socket-hmp,mode=readline \



3.login guest and check disks

Actual results:
Failed on step2 with error message:
Could not open '/home/qsd/vhost1.sock': No such device or address



Expected results:
boot succeed

Additional info:

Comment 1 Stefano Garzarella 2023-07-25 07:56:29 UTC
I'm reducing the severity since we don't have any layered product that will use virtio-blk-vhost-user for now.

I sent upstream some changes for QEMU and libblkio.
QEMU series here: https://lore.kernel.org/qemu-devel/20230724154611.178858-1-sgarzare@redhat.com/
libblkio MR here: https://gitlab.com/libblkio/libblkio/-/merge_requests/208

The first patch of the QEMU series should fix this issue, the other changes (second patch of the QEMU series and libblkio MR) are more for future changes.

Comment 5 Yanan Fu 2023-08-10 11:18:14 UTC
QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass.

Comment 8 qing.wang 2023-08-14 02:48:28 UTC
Pass test multi disks booting, hotplug, options, installation test on

Red Hat Enterprise Linux release 9.3 Beta (Plow)
5.14.0-352.el9.x86_64
qemu-kvm-8.0.0-11.el9.x86_64
seabios-bin-1.16.1-1.el9.noarch
edk2-ovmf-20230524-2.el9.noarch
libvirt-9.5.0-5.el9.x86_64
virtio-win-prewhql-0.1-240.iso


qemu-storage-daemon \
 --chardev socket,path=/home/qsd/qmp.sock,server=on,wait=off,id=char1 \
 --monitor chardev=char1 \
 --blockdev file,filename=/home/kvm_autotest_root/images/qsd1.qcow2,node-name=prot1 \
 --blockdev qcow2,file=prot1,node-name=disk1 \
 --export vhost-user-blk,id=exp1,addr.type=unix,addr.path=/home/qsd/vhost1.sock,node-name=disk1,writable=on \
 --blockdev file,filename=/home/kvm_autotest_root/images/qsd2.qcow2,node-name=prot2 \
 --blockdev qcow2,file=prot2,node-name=disk2 \
 --export vhost-user-blk,id=exp2,addr.type=unix,addr.path=/home/qsd/vhost2.sock,node-name=disk2,writable=on \
 --blockdev file,filename=/home/kvm_autotest_root/images/qsd3.qcow2,node-name=prot3 \
 --blockdev qcow2,file=prot3,node-name=disk3 \
 --export vhost-user-blk,id=exp3,addr.type=unix,addr.path=/home/qsd/install.sock,node-name=disk3,writable=on \
 --blockdev file,filename=/home/kvm_autotest_root/images/qsd4.qcow2,node-name=prot4 \
 --blockdev qcow2,file=prot4,node-name=disk4 \
 --export vhost-user-blk,id=exp4,addr.type=unix,addr.path=/home/qsd/vhost4.sock,node-name=disk4,writable=on


/usr/libexec/qemu-kvm \
 -name testvm \
 -machine q35,memory-backend=mem \
 -object memory-backend-memfd,id=mem,size=6G,share=on \
 -m 6G \
 -smp 2 \
 -cpu host,+kvm_pv_unhalt \
 -device ich9-usb-ehci1,id=usb1 \
 -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
 -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x3,chassis=1 \
 -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x3.0x1,bus=pcie.0,chassis=2 \
 -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x3.0x2,bus=pcie.0,chassis=3 \
 -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x3.0x3,bus=pcie.0,chassis=4 \
 -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x3.0x4,bus=pcie.0,chassis=5 \
 -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x3.0x5,bus=pcie.0,chassis=6 \
 -device pcie-root-port,id=pcie-root-port-6,port=0x6,addr=0x3.0x6,bus=pcie.0,chassis=7 \
 -device pcie-root-port,id=pcie-root-port-7,port=0x7,addr=0x3.0x7,bus=pcie.0,chassis=8 \
 -device pcie-root-port,id=pcie_extra_root_port_0,bus=pcie.0,addr=0x4 \
 -object iothread,id=iothread0 \
 -device virtio-scsi-pci,id=scsi0,bus=pcie-root-port-0,iothread=iothread0 \
 -blockdev driver=qcow2,file.driver=file,cache.direct=off,cache.no-flush=on,file.filename=/home/kvm_autotest_root/images/rhel930-64-virtio-scsi.qcow2,node-name=drive_image1,file.aio=threads \
 -device scsi-hd,id=os,drive=drive_image1,bus=scsi0.0,bootindex=0,serial=OS_DISK \
 -blockdev node-name=prot_stg0,driver=virtio-blk-vhost-user,path=/home/qsd/vhost1.sock,cache.direct=on,discard=unmap,detect-zeroes=on \
 -blockdev node-name=fmt_stg0,driver=raw,file=prot_stg0 \
 -device virtio-blk-pci,iothread=iothread0,share-rw=on,serial=data0,bus=pcie-root-port-4,addr=0,id=stg0,drive=fmt_stg0,bootindex=1 \
 -blockdev node-name=prot_stg1,driver=virtio-blk-vhost-user,path=/home/qsd/vhost4.sock,cache.direct=on,auto-read-only=on,force-share=off,read-only=on \
 -blockdev node-name=fmt_stg1,driver=raw,file=prot_stg1,read-only=on \
 -device scsi-hd,id=stg1,drive=fmt_stg1,serial=data1,bootindex=2 \
 -vnc :5 \
 -monitor stdio \
 -qmp tcp:0:5955,server=on,wait=off \
 -device virtio-net-pci,mac=9a:b5:b6:b1:b2:b7,id=nic1,netdev=nicpci,bus=pcie-root-port-7 \
 -netdev tap,id=nicpci \
 -boot menu=on,reboot-timeout=1000,strict=off \
 -chardev socket,id=socket-serial,path=/var/tmp/socket-serial,logfile=/var/tmp/file-serial.log,mux=on,server=on,wait=off \
 -serial chardev:socket-serial \
 -chardev file,path=/var/tmp/file-bios.log,id=file-bios \
 -device isa-debugcon,chardev=file-bios,iobase=0x402 \
 -chardev socket,id=socket-qmp,path=/var/tmp/socket-qmp,logfile=/var/tmp/file-qmp.log,mux=on,server=on,wait=off \
 -mon chardev=socket-qmp,mode=control \
 -chardev socket,id=socket-hmp,path=/var/tmp/socket-hmp,logfile=/var/tmp/file-hmp.log,mux=on,server=on,wait=off \
 -mon chardev=socket-hmp,mode=readline

Comment 10 errata-xmlrpc 2023-11-07 08:28:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: qemu-kvm security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6368