Bug 2213317
| Summary: | Enable libblkio-based block drivers in QEMU | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Stefan Hajnoczi <stefanha> |
| Component: | qemu-kvm | Assignee: | Stefan Hajnoczi <stefanha> |
| qemu-kvm sub component: | Storage | QA Contact: | qing.wang <qinwang> |
| Status: | CLOSED ERRATA | Docs Contact: | |
| Severity: | unspecified | ||
| Priority: | unspecified | CC: | aliang, coli, jinzhao, jjongsma, juzhang, kwolf, mrezanin, qinwang, rjones, sgarzare, vgoyal, virt-maint |
| Version: | 9.3 | Keywords: | FutureFeature, Triaged |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | qemu-kvm-8.0.0-11.el9 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-11-07 08:27:35 UTC | Type: | Feature Request |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | 2225354 | ||
| Bug Blocks: | 1886123, 2180076 | ||
QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass. Failed Test on Red Hat Enterprise Linux release 9.3 Beta (Plow) 5.14.0-332.el9.x86_64 qemu-kvm-8.0.0-6.el9.x86_64 seabios-bin-1.16.1-1.el9.noarch edk2-ovmf-20230301gitf80f052277c8-5.el9.noarch virtio-win-prewhql-0.1-238.iso steps 1. create vhost vdpa device modprobe -r vhost-vdpa modprobe -r vdpa-sim-blk modprobe vhost-vdpa modprobe vdpa-sim-blk vdpa dev add mgmtdev vdpasim_blk name blk0 vdpa dev add mgmtdev vdpasim_blk name blk1 vdpa dev list -jp ls /dev/vhost-vdpa* [ $? -ne 0 ] && echo "wrong create vdpa device" 2. boot vm with driver /usr/libexec/qemu-kvm \ -name testvm \ -machine q35 \ -m 6G \ -smp 2 \ -cpu host,+kvm_pv_unhalt \ -device ich9-usb-ehci1,id=usb1 \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ \ \ -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x3,chassis=1 \ -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x3.0x1,bus=pcie.0,chassis=2 \ -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x3.0x2,bus=pcie.0,chassis=3 \ -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x3.0x3,bus=pcie.0,chassis=4 \ -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x3.0x4,bus=pcie.0,chassis=5 \ -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x3.0x5,bus=pcie.0,chassis=6 \ -device pcie-root-port,id=pcie-root-port-6,port=0x6,addr=0x3.0x6,bus=pcie.0,chassis=7 \ -device pcie-root-port,id=pcie-root-port-7,port=0x7,addr=0x3.0x7,bus=pcie.0,chassis=8 \ -device pcie-root-port,id=pcie_extra_root_port_0,bus=pcie.0,addr=0x4 \ -object iothread,id=iothread0 \ -device virtio-scsi-pci,id=scsi0,bus=pcie-root-port-0 \ -blockdev driver=qcow2,file.driver=file,cache.direct=off,cache.no-flush=on,file.filename=/home/kvm_autotest_root/images/rhel930-64-virtio-scsi.qcow2,node-name=drive_image1,file.aio=threads \ -device scsi-hd,id=os,drive=drive_image1,bus=scsi0.0,bootindex=0,serial=OS_DISK \ \ -blockdev node-name=file_stg1,driver=virtio-blk-vhost-vdpa,path=/dev/vhost-vdpa-0,cache.direct=on,cache.no-flush=off,discard=unmap \ -blockdev node-name=drive_stg1,driver=raw,cache.direct=on,cache.no-flush=off,file=file_stg1 \ -device virtio-blk-pci,iothread=iothread0,serial=stg1,bus=pcie.0-root-port-4,addr=0x0,write-cache=on,id=stg1,drive=drive_stg1,rerror=report,werror=report \ \ -blockdev node-name=file_stg2,driver=virtio-blk-vhost-vdpa,path=/dev/vhost-vdpa-1,cache.direct=on,cache.no-flush=off,discard=unmap \ -blockdev node-name=drive_stg2,driver=raw,cache.direct=on,cache.no-flush=off,file=file_stg2 \ -device scsi-hd,id=stg2,drive=drive_stg2,serial=stg2 \ -vnc :5 \ -monitor stdio \ -qmp tcp:0:5955,server=on,wait=off \ -device virtio-net-pci,mac=9a:b5:b6:b1:b2:b7,id=nic1,netdev=nicpci,bus=pcie-root-port-7 \ -netdev tap,id=nicpci \ -boot menu=on,reboot-timeout=1000,strict=off \ \ It is failed qemu-kvm: -blockdev node-name=file_stg1,driver=virtio-blk-vhost-vdpa,path=/dev/vhost-vdpa-0,cache.direct=on,cache.no-flush=off,discard=unmap: Unknown driver 'virtio-blk-vhost-vdpa' (In reply to Stefano Garzarella from comment #10) > (In reply to qing.wang from comment #7) > > Failed Test on > > > > Red Hat Enterprise Linux release 9.3 Beta (Plow) > > 5.14.0-332.el9.x86_64 > > qemu-kvm-8.0.0-6.el9.x86_64 > > seabios-bin-1.16.1-1.el9.noarch > > edk2-ovmf-20230301gitf80f052277c8-5.el9.noarch > > virtio-win-prewhql-0.1-238.iso > > > @qinwang maybe you need to install also > qemu-kvm-block-blkio-8.0.0-6.el9.x86_64 > > IIUC blkio drivers are built in a QEMU module it already is installed. root@dell-per440-07 /home/kar/tp-qemu/qemu/tests $ rpm -qa|grep qemu-kvm-block-blkio qemu-kvm-block-blkio-8.0.0-6.el9.x86_64 qemu-kvm-block-blkio-debuginfo-8.0.0-6.el9.x86_64 root@dell-per440-07 /home/kar/tp-qemu/qemu/tests $ rpm -qa|grep libblkio libblkio-1.3.0-1.el9.x86_64 Hi I think the problem is in block/meson.build and scripts/modules/module_block.py.
scripts/modules/module_block.py is supposed to fill `block_driver_modules[]` array, but we need to update it to handle libblkio drivers in the right way.
Current block_driver_modules[] doesn't include them:
(gdb) p block_driver_modules
$30 = {{format_name = 0x55555605c69d <str.67.llvm> "http",
protocol_name = 0x55555605c69d <str.67.llvm> "http",
library_name = 0x55555608c68f <str.155.llvm> "curl"}, {
format_name = 0x55555605c6a6 <str.68.llvm> "https",
protocol_name = 0x55555605c6a6 <str.68.llvm> "https",
library_name = 0x55555608c68f <str.155.llvm> "curl"}, {
format_name = 0x55555605dbd9 <str.62.llvm> "ftp", protocol_name = 0x55555605dbd9 <str.62.llvm> "ftp",
library_name = 0x55555608c68f <str.155.llvm> "curl"}, {
format_name = 0x55555605c671 <str.63.llvm> "ftps",
protocol_name = 0x55555605c671 <str.63.llvm> "ftps",
library_name = 0x55555608c68f <str.155.llvm> "curl"}, {
format_name = 0x55555608c69a <str.3.llvm> "rbd", protocol_name = 0x55555608c69a <str.3.llvm> "rbd",
library_name = 0x55555608c69a <str.3.llvm> "rbd"}}
We also need to fix a typo in .distro/qemu-kvm.spec: s/virtio-blk-vdpa-blk/virtio-blk-vhost-vdpa Thanks, Stefano. I will send patches. Module loading fix posted upstream: https://lore.kernel.org/qemu-devel/20230703103524.162855-1-stefanha@redhat.com/T/#u I will fix the qemu-kvm.spec typo in the backport. QE bot(pre verify): Set 'Verified:Tested,SanityOnly' as gating/tier1 test pass. VM can not boot, it hangs on "Probing EDD (edd=off to disable)..." Red Hat Enterprise Linux release 9.3 Beta (Plow) 5.14.0-332.el9.x86_64 qemu-kvm-8.0.0-8.el9.x86_64 seabios-bin-1.16.1-1.el9.noarch edk2-ovmf-20230524-2.el9.noarch /usr/libexec/qemu-kvm \ -name testvm \ -machine q35 \ -m 6G \ -smp 2 \ -cpu host,+kvm_pv_unhalt \ -device ich9-usb-ehci1,id=usb1 \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ \ \ -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x3,chassis=1 \ -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x3.0x1,bus=pcie.0,chassis=2 \ -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x3.0x2,bus=pcie.0,chassis=3 \ -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x3.0x3,bus=pcie.0,chassis=4 \ -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x3.0x4,bus=pcie.0,chassis=5 \ -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x3.0x5,bus=pcie.0,chassis=6 \ -device pcie-root-port,id=pcie-root-port-6,port=0x6,addr=0x3.0x6,bus=pcie.0,chassis=7 \ -device pcie-root-port,id=pcie-root-port-7,port=0x7,addr=0x3.0x7,bus=pcie.0,chassis=8 \ -device pcie-root-port,id=pcie_extra_root_port_0,bus=pcie.0,addr=0x4 \ -object iothread,id=iothread0 \ -device virtio-scsi-pci,id=scsi0,bus=pcie-root-port-0 \ -blockdev driver=qcow2,file.driver=file,cache.direct=off,cache.no-flush=on,file.filename=/home/kvm_autotest_root/images/rhel930-64-virtio-scsi.qcow2,node-name=drive_image1,file.aio=threads \ -device scsi-hd,id=os,drive=drive_image1,bus=scsi0.0,bootindex=0,serial=OS_DISK \ \ -blockdev node-name=prot_stg1,driver=virtio-blk-vhost-vdpa,path=/dev/vhost-vdpa-0,cache.direct=on \ -blockdev node-name=fmt_stg1,driver=raw,file=prot_stg1 \ -device virtio-blk-pci,bus=pcie-root-port-4,addr=0,id=stg1,drive=fmt_stg1 \ -vnc :5 \ -monitor stdio \ -qmp tcp:0:5955,server=on,wait=off \ -device virtio-net-pci,mac=9a:b5:b6:b1:b2:b7,id=nic1,netdev=nicpci,bus=pcie-root-port-7 \ -netdev tap,id=nicpci \ -boot menu=on,reboot-timeout=1000,strict=off \ \ Do you know what is wrong with my command line ? Can you interrupt grub boot, and then edit the boot command line. - Remove anything that says "rhgb quiet" - Add "debug" That might give more information about where it's really hanging. If possible it'd be nice if you could attach a virtual serial console, as that will help to capture the full boot messages. (In reply to Richard W.M. Jones from comment #24) > Can you interrupt grub boot, and then edit the boot command line. > > - Remove anything that says "rhgb quiet" > > - Add "debug" > > That might give more information about where it's really hanging. > > If possible it'd be nice if you could attach a virtual serial console, as > that will help to capture the full boot messages. It hangs on the very beginning stage, no debug info display http://fileshare.hosts.qa.psi.pek2.redhat.com/pub/section2/images_backup/qbugs/2213317/2023-07-19/boot.png I understand that. However the EDD message is typically not where the hang actually happens in my experience. I would strongly suggest attaching a serial console, then editing the Linux command line (with grub) to remove "rhgb quiet", and add "debug console=/dev/ttyS0", then see what is printed on the serial console. (In reply to Richard W.M. Jones from comment #26) > I understand that. However the EDD message is typically not where the hang > actually > happens in my experience. > > I would strongly suggest attaching a serial console, then editing the Linux > command > line (with grub) to remove "rhgb quiet", and add "debug console=/dev/ttyS0", > then see > what is printed on the serial console. the console is added in kernel parameter, http://fileshare.hosts.qa.psi.pek2.redhat.com/pub/section2/images_backup/qbugs/2213317/2023-07-20/grub.png No log in the serial log /usr/libexec/qemu-kvm \ -name testvm \ -machine q35 \ -m 6G \ -smp 2 \ -cpu host,+kvm_pv_unhalt \ -device ich9-usb-ehci1,id=usb1 \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ \ \ -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x3,chassis=1 \ -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x3.0x1,bus=pcie.0,chassis=2 \ -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x3.0x2,bus=pcie.0,chassis=3 \ -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x3.0x3,bus=pcie.0,chassis=4 \ -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x3.0x4,bus=pcie.0,chassis=5 \ -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x3.0x5,bus=pcie.0,chassis=6 \ -device pcie-root-port,id=pcie-root-port-6,port=0x6,addr=0x3.0x6,bus=pcie.0,chassis=7 \ -device pcie-root-port,id=pcie-root-port-7,port=0x7,addr=0x3.0x7,bus=pcie.0,chassis=8 \ -device pcie-root-port,id=pcie_extra_root_port_0,bus=pcie.0,addr=0x4 \ -object iothread,id=iothread0 \ -device virtio-scsi-pci,id=scsi0,bus=pcie-root-port-0 \ -blockdev driver=qcow2,file.driver=file,cache.direct=off,cache.no-flush=on,file.filename=/home/kvm_autotest_root/images/rhel930-64-virtio-scsi.qcow2,node-name=drive_image1,file.aio=threads \ -device scsi-hd,id=os,drive=drive_image1,bus=scsi0.0,bootindex=0,serial=OS_DISK \ \ -blockdev node-name=prot_stg1,driver=virtio-blk-vhost-vdpa,path=/dev/vhost-vdpa-0,cache.direct=on \ -blockdev node-name=fmt_stg1,driver=raw,file=prot_stg1 \ -device virtio-blk-pci,bus=pcie-root-port-4,addr=0,id=stg1,drive=fmt_stg1,bootindex=2 \ -vnc :5 \ -monitor stdio \ -qmp tcp:0:5955,server=on,wait=off \ -device virtio-net-pci,mac=9a:b5:b6:b1:b2:b7,id=nic1,netdev=nicpci,bus=pcie-root-port-7 \ -netdev tap,id=nicpci \ -boot menu=on,reboot-timeout=1000,strict=off \ \ -chardev socket,id=socket-serial,path=/var/tmp/socket-serial,logfile=/var/tmp/file-serial.log,mux=on,server=on,wait=off \ -serial chardev:socket-serial \ -chardev file,path=/var/tmp/file-bios.log,id=file-bios \ -device isa-debugcon,chardev=file-bios,iobase=0x402 \ root@dell-per6515-03 /home $ cat -A /var/tmp/file-serial.log ^[[H^[[J^[[1;1H^[[?25l^[[m^[[H^[[J^[[1;1H^[[2;32HGRUB version 2.06$ ^M$ ^M^[[m^[[4;2H+----------------------------------------------------------------------------+^[[5;2H|^[[5;79H|^[[6;2H|^[[6;79H|^[[7;2H|^[[7;79H|^[[8;2H|^[[8;79H|^[[9;2H|^[[9;79H|^[[10;2H|^[[10;79H|^[[11;2H|^[[11;79H|^[[12;2H|^[[12;79H|^[[13;2H|^[[13;79H|^[[14;2H|^[[14;79H|^[[15;2H|^[[15;79H|^[[16;2H|^[[16;79H|^[[17;2H+----------------------------------------------------------------------------+^[[m^[[18;2H^[[19;2H^[[m Use the ^ and v keys to select which entry is highlighted. $ ^M Press enter to boot the selected OS, `e' to edit the commands $ ^M before booting or `c' for a command-line. ^[[5;80H ^[[7m^[[5;3H*Red Hat Enterprise Linux (5.14.0-332.el9.x86_64) 9.3 (Plow) ^[[m^[[5;78H^[[m^[[m^[[6;3H Red Hat Enterprise Linux (0-rescue-df1ed2684a0341ee8eb28b5f64e0bc73) 9.3 (>^[[m^[[6;78H^[[m^[[m^[[7;3H ^[[m^[[7;78H^[[m^[[m^[[8;3H ^[[m^[[8;78H^[[m^[[m^[[9;3H ^[[m^[[9;78H^[[m^[[m^[[10;3H ^[[m^[[10;78H^[[m^[[m^[[11;3H ^[[m^[[11;78H^[[m^[[m^[[12;3H ^[[m^[[12;78H^[[m^[[m^[[13;3H ^[[m^[[13;78H^[[m^[[m^[[14;3H ^[[m^[[14;78H^[[m^[[m^[[15;3H ^[[m^[[15;78H^[[m^[[m^[[16;3H ^[[m^[[16;78H^[[m^[[16;80H ^[[5;78H^[[22;1H The highlighted entry will be executed automatically in 5s. ^[[5;78H^[[22;1H The highlighted entry will be executed automatically in 4s. ^[[5;78H^[[22;1H The highlighted entry will be executed automatically in 3s. ^[[5;78H^[[22;1H The highlighted entry will be executed automatically in 2s. ^[[5;78H^[[22;1H The highlighted entry will be executed automatically in 1s. ^[[5;78H^[[22;1H The highlighted entry will be executed automatically in 0s. ^[[5;78H^[[?25h^[[H^[[J^[[1;1H^[[H^[[J^[[1;1H Booting `Red Hat Enterprise Linux (5.14.0-332.el9.x86_64) 9.3 (Plow)'$ ^M$ The kernel printed nothing at all on the serial port, not even the EDD message, so there's obviously something wrong with your serial port configuration. I'd try to fix that first so you can get the all important kernel messages out, otherwise this bug is going to be impossible to debug. Qing shared the access with the machine and I think the problem is that vhost-vdpa devices still require pinning all the guest pages.
So using 4k pages with a 6G guest ram, can take a lot of time.
This requirement may disappear if we will backport commit 4bb94d2de2fa ("vdpa_sim: add support for user VA") [1] for the vDPA simulators but in general HW device don't support this yet, so for now my advice is to use hugepages with vhost-vdpa.
Something like this:
# allocate 3100 hugepages (2M): 6G/2M = 3072 + some extra pages
echo 3100 > /proc/sys/vm/nr_hugepages
/usr/libexec/qemu-kvm \
...
-machine q35,memory-backend=mem \
-object memory-backend-file,share=on,id=mem,size=6G,mem-path=/dev/hugepages/libvirt/qemu \
Or as Qing suggested a memory-backend-memfd since Linux should support transparent huge-pages of shmem/memfd memory:
/usr/libexec/qemu-kvm \
...
-machine q35,memory-backend=mem \
-object memory-backend-memfd,id=mem,size=6G,share=on \
We also need to increase memlock limits in /etc/security/limits.conf:
* soft memlock unlimited
* hard memlock unlimited
[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4bb94d2de2fa90aa8d4e1ce940b1b2f7708cf141
VM can not boot with qemu-storage-daemon \ --chardev socket,path=/home/qsd/qmp.sock,server=on,wait=off,id=char1 \ --monitor chardev=char1 \ --blockdev file,filename=/home/kvm_autotest_root/images/qsd1.qcow2,node-name=prot1 \ --blockdev qcow2,file=prot1,node-name=disk1 \ --export vhost-user-blk,id=exp1,addr.type=unix,addr.path=/home/qsd/vhost1.sock,node-name=disk1,writable=on \ --blockdev file,filename=/home/kvm_autotest_root/images/qsd2.qcow2,node-name=prot2 \ --blockdev qcow2,file=prot2,node-name=disk2 \ --export vhost-user-blk,id=exp2,addr.type=unix,addr.path=/home/qsd/vhost2.sock,node-name=disk2,writable=on \ 2.Boot VM with virtio-blk-vhost-userdriver /usr/libexec/qemu-kvm \ -name testvm \ -machine q35,memory-backend=mem \ -object memory-backend-memfd,id=mem,size=6G,share=on \ -m 6G \ -smp 2 \ -cpu host,+kvm_pv_unhalt \ -device ich9-usb-ehci1,id=usb1 \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ \ \ -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x3,chassis=1 \ -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x3.0x1,bus=pcie.0,chassis=2 \ -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x3.0x2,bus=pcie.0,chassis=3 \ -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x3.0x3,bus=pcie.0,chassis=4 \ -device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x3.0x4,bus=pcie.0,chassis=5 \ -device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x3.0x5,bus=pcie.0,chassis=6 \ -device pcie-root-port,id=pcie-root-port-6,port=0x6,addr=0x3.0x6,bus=pcie.0,chassis=7 \ -device pcie-root-port,id=pcie-root-port-7,port=0x7,addr=0x3.0x7,bus=pcie.0,chassis=8 \ -device pcie-root-port,id=pcie_extra_root_port_0,bus=pcie.0,addr=0x4 \ -object iothread,id=iothread0 \ -device virtio-scsi-pci,id=scsi0,bus=pcie-root-port-0,iothread=iothread0 \ -blockdev driver=qcow2,file.driver=file,cache.direct=off,cache.no-flush=on,file.filename=/home/kvm_autotest_root/images/rhel930-64-virtio-scsi.qcow2,node-name=drive_image1,file.aio=threads \ -device scsi-hd,id=os,drive=drive_image1,bus=scsi0.0,bootindex=0,serial=OS_DISK \ \ -blockdev node-name=prot_stg0,driver=virtio-blk-vhost-user,path=/home/qsd/vhost1.sock,cache.direct=on \ -blockdev node-name=fmt_stg0,driver=raw,file=prot_stg0 \ -device virtio-blk-pci,iothread=iothread0,bus=pcie-root-port-4,addr=0,id=stg0,drive=fmt_stg0,bootindex=1 \ \ -blockdev node-name=prot_stg1,driver=virtio-blk-vhost-user,path=/home/qsd/vhost2.sock,cache.direct=on \ -blockdev node-name=fmt_stg1,driver=raw,file=prot_stg1 \ -device scsi-hd,id=stg1,drive=fmt_stg1,bootindex=2 \ -vnc :5 \ -monitor stdio \ -qmp tcp:0:5955,server=on,wait=off \ -device virtio-net-pci,mac=9a:b5:b6:b1:b2:b7,id=nic1,netdev=nicpci,bus=pcie-root-port-7 \ -netdev tap,id=nicpci \ -boot menu=on,reboot-timeout=1000,strict=off \ \ -chardev socket,id=socket-serial,path=/var/tmp/socket-serial,logfile=/var/tmp/file-serial.log,mux=on,server=on,wait=off \ -serial chardev:socket-serial \ -chardev file,path=/var/tmp/file-bios.log,id=file-bios \ -device isa-debugcon,chardev=file-bios,iobase=0x402 \ \ -chardev socket,id=socket-qmp,path=/var/tmp/socket-qmp,logfile=/var/tmp/file-qmp.log,mux=on,server=on,wait=off \ -mon chardev=socket-qmp,mode=control \ -chardev socket,id=socket-hmp,path=/var/tmp/socket-hmp,logfile=/var/tmp/file-hmp.log,mux=on,server=on,wait=off \ -mon chardev=socket-hmp,mode=readline \ Boot failed : qemu-kvm: -blockdev node-name=prot_stg0,driver=virtio-blk-vhost-user,path=/home/qsd/vhost1.sock,cache.direct=on: Could not open '/home/qsd/vhost1.sock': No such device or address The driver virtio-blk-vhost-vdpa works on
Red Hat Enterprise Linux release 9.3 Beta (Plow)
5.14.0-340.el9.x86_64
qemu-kvm-8.0.0-8.el9.x86_64
seabios-bin-1.16.1-1.el9.noarch
edk2-ovmf-20230524-2.el9.noarch
libvirt-9.3.0-2.el9.x86_64
Test steps:
1. parare vhost vdpa disks on host
modprobe vhost-vdpa
modprobe vdpa-sim-blk
vdpa dev add mgmtdev vdpasim_blk name blk0
vdpa dev add mgmtdev vdpasim_blk name blk1
vdpa dev list -jp
ls /dev/vhost-vdpa*
[ $? -ne 0 ] && echo "wrong create vdpa device"
2. boot VM with virtio-blk-vhost-vdpa driver
/usr/libexec/qemu-kvm \
-name testvm \
-machine q35,memory-backend=mem \
-object memory-backend-memfd,id=mem,size=6G,share=on \
-m 6G \
-smp 2 \
-cpu host,+kvm_pv_unhalt \
-device ich9-usb-ehci1,id=usb1 \
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \
\
\
-device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x3,chassis=1 \
-device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x3.0x1,bus=pcie.0,chassis=2 \
-device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x3.0x2,bus=pcie.0,chassis=3 \
-device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x3.0x3,bus=pcie.0,chassis=4 \
-device pcie-root-port,id=pcie-root-port-4,port=0x4,addr=0x3.0x4,bus=pcie.0,chassis=5 \
-device pcie-root-port,id=pcie-root-port-5,port=0x5,addr=0x3.0x5,bus=pcie.0,chassis=6 \
-device pcie-root-port,id=pcie-root-port-6,port=0x6,addr=0x3.0x6,bus=pcie.0,chassis=7 \
-device pcie-root-port,id=pcie-root-port-7,port=0x7,addr=0x3.0x7,bus=pcie.0,chassis=8 \
-device pcie-root-port,id=pcie_extra_root_port_0,bus=pcie.0,addr=0x4 \
-object iothread,id=iothread0 \
-device virtio-scsi-pci,id=scsi0,bus=pcie-root-port-0,iothread=iothread0 \
-blockdev driver=qcow2,file.driver=file,cache.direct=off,cache.no-flush=on,file.filename=/home/kvm_autotest_root/images/rhel930-64-virtio-scsi.qcow2,node-name=drive_image1,file.aio=threads \
-device scsi-hd,id=os,drive=drive_image1,bus=scsi0.0,bootindex=0,serial=OS_DISK \
\
-blockdev node-name=prot_stg0,driver=virtio-blk-vhost-vdpa,path=/dev/vhost-vdpa-0,cache.direct=on \
-blockdev node-name=fmt_stg0,driver=raw,file=prot_stg0 \
-device virtio-blk-pci,iothread=iothread0,bus=pcie-root-port-4,addr=0,id=stg0,drive=fmt_stg0,bootindex=1 \
\
-blockdev node-name=prot_stg1,driver=virtio-blk-vhost-vdpa,path=/dev/vhost-vdpa-1,cache.direct=on \
-blockdev node-name=fmt_stg1,driver=raw,file=prot_stg1 \
-device scsi-hd,id=stg1,drive=fmt_stg1,bootindex=2 \
-vnc :5 \
-monitor stdio \
-qmp tcp:0:5955,server=on,wait=off \
-device virtio-net-pci,mac=9a:b5:b6:b1:b2:b7,id=nic1,netdev=nicpci,bus=pcie-root-port-7 \
-netdev tap,id=nicpci \
-boot menu=on,reboot-timeout=1000,strict=off \
\
-chardev socket,id=socket-serial,path=/var/tmp/socket-serial,logfile=/var/tmp/file-serial.log,mux=on,server=on,wait=off \
-serial chardev:socket-serial \
-chardev file,path=/var/tmp/file-bios.log,id=file-bios \
-device isa-debugcon,chardev=file-bios,iobase=0x402 \
\
-chardev socket,id=socket-qmp,path=/var/tmp/socket-qmp,logfile=/var/tmp/file-qmp.log,mux=on,server=on,wait=off \
-mon chardev=socket-qmp,mode=control \
-chardev socket,id=socket-hmp,path=/var/tmp/socket-hmp,logfile=/var/tmp/file-hmp.log,mux=on,server=on,wait=off \
-mon chardev=socket-hmp,mode=readline \
3. login guest and execute simple io
cat guest_io.sh
function tests_failed() {
exit_code="$?"
echo "Test failed: $1"
exit "${exit_code}"
}
vdpa_devs=`lsblk -nd|grep 128M|awk '{print $1}'`
echo ${vdpa_devs}
for dev in ${vdpa_devs};do
echo "$dev"
mkfs.xfs -f /dev/${dev} || tests_failed "format"
mkdir -p /home/${dev}
mount /dev/${dev} /home/${dev} || tests_failed "mount"
dd if=/dev/zero of=/home/${dev}/test.img count=100 bs=1M oflag=direct || tests_failed "IO"
umount -fl /home/${dev}
done
4.unplug disks
{"execute": "device_del", "arguments": {"id": "stg1"}}
{"execute": "blockdev-del","arguments": {"node-name": "fmt_stg1"}}
{"execute": "blockdev-del","arguments": {"node-name":"prot_stg1"}}
{"execute": "device_del", "arguments": {"id": "stg0"}}
{"execute": "blockdev-del","arguments": {"node-name": "fmt_stg0"}}
{"execute": "blockdev-del","arguments": {"node-name":"prot_stg0"}}
5. check disks in guest, the disks should disappear
6. plug disks
{"execute": "blockdev-add", "arguments": {"node-name": "prot_stg0", "driver": "virtio-blk-vhost-vdpa", "path": "/dev/vhost-vdpa-0","cache": {"direct": true, "no-flush": false}}}
{"execute": "blockdev-add", "arguments": {"node-name": "fmt_stg0", "driver": "raw", "file": "prot_stg0"}}
{"execute": "device_add", "arguments": {"driver": "virtio-blk-pci", "id": "stg0", "drive": "fmt_stg0","bus":"pcie-root-port-4"}}
{"execute": "blockdev-add", "arguments": {"node-name": "prot_stg1", "driver": "virtio-blk-vhost-vdpa", "path": "/dev/vhost-vdpa-1","cache": {"direct": true, "no-flush": false}}}
{"execute": "blockdev-add", "arguments": {"node-name": "fmt_stg1", "driver": "raw", "file": "prot_stg1"}}
{"execute": "device_add", "arguments": {"driver": "scsi-hd", "id": "stg1", "drive": "fmt_stg1"}}
7.check disks in guest, the disks should exist
8. run io like step 3
(In reply to qing.wang from comment #30) > VM can not boot with [...] > Boot failed : > qemu-kvm: -blockdev > node-name=prot_stg0,driver=virtio-blk-vhost-user,path=/home/qsd/vhost1.sock, > cache.direct=on: Could not open '/home/qsd/vhost1.sock': No such device or > address Argh, I replicated upstream. I think I introduced this issue while adding `fd` support for vhost-vdpa. In QEMU we are using blkio_get_int("fd") to check if fd passing is supported or not, but in libblkio the `fd` property is shared with all the virtio-blk drivers and the error is returned only during blkio_connect(). So we can't use `blkio_get_int("fd")` in QEMU to check if the driver supports it or not. @qinwang please open another BZ to track that issue, for now let's use vhost-vdpa or io_uring to test libblkio support in QEMU. I'll try to fix that issue in QEMU or libblkio. (In reply to Stefano Garzarella from comment #32) > (In reply to qing.wang from comment #30) > @qinwang please open another BZ to track that issue, for now > let's use vhost-vdpa or io_uring to test libblkio support in QEMU. > > I'll try to fix that issue in QEMU or libblkio. Sounds good. We can make bz#2213317 dependent on the new BZ that addresses the virtio-blk-vhost-user "Could not open '/home/qsd/vhost1.sock': No such device or address" error. (In reply to Stefan Hajnoczi from comment #33) > (In reply to Stefano Garzarella from comment #32) > > (In reply to qing.wang from comment #30) > > @qinwang please open another BZ to track that issue, for now > > let's use vhost-vdpa or io_uring to test libblkio support in QEMU. > > > > I'll try to fix that issue in QEMU or libblkio. > > Sounds good. We can make bz#2213317 dependent on the new BZ that addresses > the virtio-blk-vhost-user "Could not open '/home/qsd/vhost1.sock': No such > device or address" error. Thanks, File a new bug Bug to track virtio-blk-vhost-user issue: 2225354 - [vdpa-blk] The new driver virtio-blk-vhost-user not work in VM booting Pass test driver virtio-blk-vhost-user scenario: multi disks booting, hotplug, options, installation test test driver virt-blk-vhost-vdpa scenario: multi disks booting, hotplug, options, migration test Red Hat Enterprise Linux release 9.3 Beta (Plow) 5.14.0-352.el9.x86_64 qemu-kvm-8.0.0-11.el9.x86_64 seabios-bin-1.16.1-1.el9.noarch edk2-ovmf-20230524-2.el9.noarch libvirt-9.5.0-5.el9.x86_64 virtio-win-prewhql-0.1-240.iso Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: qemu-kvm security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:6368 |
Description of problem: The libblkio package has been added to RHEL in order to provide vdpa-blk and vhost-user-blk support to qemu-kvm and other programs. It is now possible to enable the libblkio-based block drivers in qemu-kvm. They are built as a module and the new optional sub-package will be called qemu-kvm-block-blkio. Note that libvirt currently does not have domain XML syntax for these block drivers, but <qemu:commandline> can be used to pass QEMU --blockdev options through. Version-Release number of selected component (if applicable): How reproducible: 100% Steps to Reproduce: 1. # modprobe vdpa_sim_blk 2. # modprobe vhost_vdpa 3. # vdpa dev add name vdpa-blk1 mgmtdev vdpasim_blk 4. # qemu-system-x86_64 ... \ --blockdev virtio-blk-vhost-vdpa,node-name=vdpa-blk0,path=/dev/vhost-vdpa-0,cache.direct=on \ --device virtio-blk-pci,drive=vdpa-blk0,serial=vdpa-blk0 Actual results: qemu-system-x86_64: --blockdev virtio-blk-vhost-vdpa,node-name=vdpa-blk0,path=/dev/vhost-vdpa-0,cache.direct=on: Unknown driver 'virtio-blk-vhost-vdpa' Expected results: A virtio-blk device with the serial string "vdpa-blk0" appears inside the guest (i.e. /dev/disk/by-id/virtio-vdpa-blk0 exists inside the guest). Additional info: