Description of problem: Hot-unplug multifunction disk the function number is not 0. but it makes other disks in same bus are unpluged. Version-Release number of selected component (if applicable): qemu-kvm-core-4.2.0-23.module+el8.3.0+6922+fd575af8.x86_64 qemu-kvm-common-4.2.0-23.module+el8.3.0+6922+fd575af8.x86_64 How reproducible: 100% Steps to Reproduce: 1.create image at first running qemu-img create -f qcow2 /home/kvm_autotest_root/images/stg0.qcow2 1G qemu-img create -f qcow2 /home/kvm_autotest_root/images/stg1.qcow2 1G qemu-img create -f qcow2 /home/kvm_autotest_root/images/stg2.qcow2 2G qemu-img create -f qcow2 /home/kvm_autotest_root/images/stg3.qcow2 3G qemu-img create -f qcow2 /home/kvm_autotest_root/images/stg4.qcow2 4G qemu-img create -f qcow2 /home/kvm_autotest_root/images/stg5.qcow2 5G qemu-img create -f qcow2 /home/kvm_autotest_root/images/stg6.qcow2 6G qemu-img create -f qcow2 /home/kvm_autotest_root/images/stg7.qcow2 7G qemu-img create -f qcow2 /home/kvm_autotest_root/images/stg8.qcow2 8G 2. boot vm /usr/libexec/qemu-kvm \ -name 'avocado-vt-vm1' \ -sandbox on \ -machine pc \ -nodefaults \ -device VGA,bus=pci.0,addr=0x2 \ -m 8096 \ -smp 12,maxcpus=12,cores=6,threads=1,dies=1,sockets=2 \ -device qemu-xhci,id=usb1,bus=pci.0,addr=0x3 \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -object iothread,id=iothread0 \ -object iothread,id=iothread1 \ -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/rhel820-64-virtio.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \ -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,write-cache=on,multifunction=on,bus=pci.0,addr=0x4,iothread=iothread0 \ -device virtio-net-pci,mac=9a:01:f5:18:bb:c8,id=idLAYRBi,netdev=idEsOS5t,bus=pci.0,addr=0x5 \ -netdev tap,id=idEsOS5t,vhost=on \ -blockdev node-name=file_cd1,driver=file,read-only=on,aio=threads,filename=/home/kvm_autotest_root/iso/windows/winutils.iso,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_cd1,driver=raw,read-only=on,cache.direct=on,cache.no-flush=off,file=file_cd1 \ -device ide-cd,id=cd1,drive=drive_cd1,bootindex=1,write-cache=on,bus=ide.0,unit=0 \ -vnc :5 \ -rtc base=localtime,clock=host,driftfix=slew \ -boot menu=off,order=cdn,once=c,strict=off \ -enable-kvm -monitor stdio \ -qmp tcp:0:5955,server,nowait \ 3.hotplug disk on the function 0-7 {"execute":"qmp_capabilities"} {"execute": "blockdev-add", "arguments": {"node-name": "file_stg1", "driver": "file", "aio": "threads", "filename": "/home/kvm_autotest_root/images/stg1.qcow2", "cache": {"direct": true, "no-flush": false}}, "id": "6O9nmhNX"} {"execute": "blockdev-add", "arguments": {"node-name": "drive_stg1", "driver": "qcow2", "cache": {"direct": true, "no-flush": false}, "file": "file_stg1"}, "id": "8JKczL0h"} {"execute": "blockdev-add", "arguments": {"node-name": "file_stg2", "driver": "file", "aio": "threads", "filename": "/home/kvm_autotest_root/images/stg2.qcow2", "cache": {"direct": true, "no-flush": false}}, "id": "6O9nmhNX"} {"execute": "blockdev-add", "arguments": {"node-name": "drive_stg2", "driver": "qcow2", "cache": {"direct": true, "no-flush": false}, "file": "file_stg2"}, "id": "8JKczL0h"} {"execute": "blockdev-add", "arguments": {"node-name": "file_stg3", "driver": "file", "aio": "threads", "filename": "/home/kvm_autotest_root/images/stg3.qcow2", "cache": {"direct": true, "no-flush": false}}, "id": "6O9nmhNX"} {"execute": "blockdev-add", "arguments": {"node-name": "drive_stg3", "driver": "qcow2", "cache": {"direct": true, "no-flush": false}, "file": "file_stg3"}, "id": "8JKczL0h"} {"execute": "blockdev-add", "arguments": {"node-name": "file_stg4", "driver": "file", "aio": "threads", "filename": "/home/kvm_autotest_root/images/stg4.qcow2", "cache": {"direct": true, "no-flush": false}}, "id": "6O9nmhNX"} {"execute": "blockdev-add", "arguments": {"node-name": "drive_stg4", "driver": "qcow2", "cache": {"direct": true, "no-flush": false}, "file": "file_stg4"}, "id": "8JKczL0h"} {"execute": "blockdev-add", "arguments": {"node-name": "file_stg5", "driver": "file", "aio": "threads", "filename": "/home/kvm_autotest_root/images/stg5.qcow2", "cache": {"direct": true, "no-flush": false}}, "id": "6O9nmhNX"} {"execute": "blockdev-add", "arguments": {"node-name": "drive_stg5", "driver": "qcow2", "cache": {"direct": true, "no-flush": false}, "file": "file_stg5"}, "id": "8JKczL0h"} {"execute": "blockdev-add", "arguments": {"node-name": "file_stg6", "driver": "file", "aio": "threads", "filename": "/home/kvm_autotest_root/images/stg6.qcow2", "cache": {"direct": true, "no-flush": false}}, "id": "6O9nmhNX"} {"execute": "blockdev-add", "arguments": {"node-name": "drive_stg6", "driver": "qcow2", "cache": {"direct": true, "no-flush": false}, "file": "file_stg6"}, "id": "8JKczL0h"} {"execute": "blockdev-add", "arguments": {"node-name": "file_stg7", "driver": "file", "aio": "threads", "filename": "/home/kvm_autotest_root/images/stg7.qcow2", "cache": {"direct": true, "no-flush": false}}, "id": "6O9nmhNX"} {"execute": "blockdev-add", "arguments": {"node-name": "drive_stg7", "driver": "qcow2", "cache": {"direct": true, "no-flush": false}, "file": "file_stg7"}, "id": "8JKczL0h"} {"execute": "blockdev-add", "arguments": {"node-name": "file_stg8", "driver": "file", "aio": "threads", "filename": "/home/kvm_autotest_root/images/stg8.qcow2", "cache": {"direct": true, "no-flush": false}}, "id": "tvII1jdT"} {"execute": "blockdev-add", "arguments": {"node-name": "drive_stg8", "driver": "qcow2", "cache": {"direct": true, "no-flush": false}, "file": "file_stg8"}, "id": "dBKzIcv3"} #3.2 {"execute": "device_add", "arguments": {"driver": "virtio-blk-pci", "id": "stg1", "drive": "drive_stg1", "write-cache": "on", "addr": "0x9.0x1", "bus": "pci.0", "iothread": "iothread1"}, "id": "moK7xdmS"} {"execute": "device_add", "arguments": {"driver": "virtio-blk-pci", "id": "stg2", "drive": "drive_stg2", "write-cache": "on", "addr": "0x9.0x2", "bus": "pci.0", "iothread": "iothread1"}, "id": "moK7xdmS"} {"execute": "device_add", "arguments": {"driver": "virtio-blk-pci", "id": "stg3", "drive": "drive_stg3", "write-cache": "on", "addr": "0x9.0x3", "bus": "pci.0", "iothread": "iothread1"}, "id": "moK7xdmS"} {"execute": "device_add", "arguments": {"driver": "virtio-blk-pci", "id": "stg4", "drive": "drive_stg4", "write-cache": "on", "addr": "0x9.0x4", "bus": "pci.0", "iothread": "iothread1"}, "id": "moK7xdmS"} {"execute": "device_add", "arguments": {"driver": "virtio-blk-pci", "id": "stg5", "drive": "drive_stg5", "write-cache": "on", "addr": "0x9.0x5", "bus": "pci.0", "iothread": "iothread1"}, "id": "moK7xdmS"} {"execute": "device_add", "arguments": {"driver": "virtio-blk-pci", "id": "stg6", "drive": "drive_stg6", "write-cache": "on", "addr": "0x9.0x6", "bus": "pci.0", "iothread": "iothread1"}, "id": "moK7xdmS"} {"execute": "device_add", "arguments": {"driver": "virtio-blk-pci", "id": "stg7", "drive": "drive_stg7", "write-cache": "on", "addr": "0x9.0x7", "bus": "pci.0", "iothread": "iothread1"}, "id": "moK7xdmS"} {"execute": "device_add", "arguments": {"driver": "virtio-blk-pci", "id": "stg8", "drive": "drive_stg8", "write-cache": "on", "multifunction": "on", "addr": "0x9.0x0", "bus": "pci.0", "iothread": "iothread1"}, "id": "95tsCv3C"} 4.check disk in guest 5.hotunplug function 7 disk {"execute":"device_del","arguments":{"id":"stg7"}} 6.hotunplug function 0 disk {"execute":"device_del","arguments":{"id":"stg8"}} Actual results: disk stg1-8 were unpluged in guest at step 5 Expected results: only disk7 unpluged in guest are step 5 other disk unpluged in guest after step6 Additional info: Same result for q35 mode step 2 in q35 /usr/libexec/qemu-kvm \ -name 'avocado-vt-vm1' \ -sandbox on \ -machine q35 \ -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \ -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 \ -nodefaults \ -device VGA,bus=pcie.0,addr=0x2 \ -m 7168 \ -smp 4,maxcpus=4,cores=2,threads=1,dies=1,sockets=2 \ -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \ -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/rhel820-64-virtio.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \ -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \ -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,write-cache=on,multifunction=on,bus=pcie-root-port-2,addr=0x0 \ -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \ -device virtio-net-pci,mac=9a:f4:f0:0e:dc:05,id=id4dZ4px,netdev=idnbbqEr,bus=pcie-root-port-3,addr=0x0 \ -netdev tap,id=idnbbqEr,vhost=on \ -vnc :5 \ -rtc base=localtime,clock=host,driftfix=slew \ -boot menu=off,order=cdn,once=c,strict=off \ -enable-kvm -monitor stdio \ -qmp tcp:0:5955,server,nowait \ -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,slot=4,addr=0x4,chassis=4 \ step 3.2 in q35 {"execute": "device_add", "arguments": {"driver": "virtio-blk-pci", "id": "stg1", "drive": "drive_stg1", "multifunction":"on","bus": "pcie_extra_root_port_0","addr":"0x0.1"}} {"execute": "device_add", "arguments": {"driver": "virtio-blk-pci", "id": "stg2", "drive": "drive_stg2", "multifunction":"on","bus": "pcie_extra_root_port_0","addr":"0x0.2"}} {"execute": "device_add", "arguments": {"driver": "virtio-blk-pci", "id": "stg3", "drive": "drive_stg3", "multifunction":"on","bus": "pcie_extra_root_port_0","addr":"0x0.3"}} {"execute": "device_add", "arguments": {"driver": "virtio-blk-pci", "id": "stg4", "drive": "drive_stg4", "multifunction":"on","bus": "pcie_extra_root_port_0","addr":"0x0.4"}} {"execute": "device_add", "arguments": {"driver": "virtio-blk-pci", "id": "stg5", "drive": "drive_stg5", "multifunction":"on","bus": "pcie_extra_root_port_0","addr":"0x0.5"}} {"execute": "device_add", "arguments": {"driver": "virtio-blk-pci", "id": "stg6", "drive": "drive_stg6", "multifunction":"on","bus": "pcie_extra_root_port_0","addr":"0x0.6"}} {"execute": "device_add", "arguments": {"driver": "virtio-blk-pci", "id": "stg7", "drive": "drive_stg7", "multifunction":"on","bus": "pcie_extra_root_port_0","addr":"0x0.7"}} {"execute": "device_add", "arguments": {"driver": "virtio-blk-pci", "id": "stg8", "drive": "drive_stg8", "multifunction":"on","bus": "pcie_extra_root_port_0","addr":"0x0.0"}}
(In reply to qing.wang from comment #0) > Description of problem: > > Hot-unplug multifunction disk the function number is not 0. but it makes > other disks in same bus are unpluged. > > Version-Release number of selected component (if applicable): > qemu-kvm-core-4.2.0-23.module+el8.3.0+6922+fd575af8.x86_64 > qemu-kvm-common-4.2.0-23.module+el8.3.0+6922+fd575af8.x86_64 > > How reproducible: > 100% Will this ever happen through libvirt?
needinfo to virtual disk owner for now, if should be covered by scsi function, re-needinfo to jgao pls
I think this bug is similar as https://bugzilla.redhat.com/show_bug.cgi?id=1610215 The PCI standard doesn't allow hotplugging something into an unused function of a slot that already has a device in it. So we can't reproduce it in libvirt.
Try without hotplug but coldplug: disk xml: <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/RHEL-8.3-x86_64-latest.qcow2'/> <backingStore/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/stg2.qcow2'/> <backingStore/> <target dev='vdc' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/stg3.qcow2'/> <backingStore/> <target dev='vdd' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x2'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/stg6.qcow2'/> <backingStore/> <target dev='vdf' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x3' multifunction='on'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/lib/libvirt/images/stg7.qcow2'/> <backingStore/> <target dev='vdh' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x4'/> </disk> [root@jgao-test1 ~]# virsh start demo Domain demo started [root@jgao-test1 ~]# virsh domblklist demo Target Source ---------------------------------------------------------------- vda /var/lib/libvirt/images/RHEL-8.3-x86_64-latest.qcow2 vdc /var/lib/libvirt/images/stg2.qcow2 vdd /var/lib/libvirt/images/stg3.qcow2 vdf /var/lib/libvirt/images/stg6.qcow2 vdh /var/lib/libvirt/images/stg7.qcow2 [root@jgao-test1 ~]# virsh detach-disk demo vdf error: No disk found whose source path or target is vdf ------>> detach failed but all disks all losed [root@jgao-test1 ~]# virsh domblklist demo Target Source ---------------------------------------------------------------- vda /var/lib/libvirt/images/RHEL-8.3-x86_64-latest.qcow2 Are these expected result for you ?
It looks like have same result on libvirt.
(In reply to gaojianan from comment #5) > Try without hotplug but coldplug: > disk xml: > <disk type='file' device='disk'> > <driver name='qemu' type='qcow2'/> > <source file='/var/lib/libvirt/images/RHEL-8.3-x86_64-latest.qcow2'/> > <backingStore/> > <target dev='vda' bus='virtio'/> > <address type='pci' domain='0x0000' bus='0x01' slot='0x00' > function='0x0'/> > </disk> > <disk type='file' device='disk'> > <driver name='qemu' type='qcow2'/> > <source file='/var/lib/libvirt/images/stg2.qcow2'/> > <backingStore/> > <target dev='vdc' bus='virtio'/> > <address type='pci' domain='0x0000' bus='0x04' slot='0x00' > function='0x1'/> > </disk> > <disk type='file' device='disk'> > <driver name='qemu' type='qcow2'/> > <source file='/var/lib/libvirt/images/stg3.qcow2'/> > <backingStore/> > <target dev='vdd' bus='virtio'/> > <address type='pci' domain='0x0000' bus='0x06' slot='0x00' > function='0x2'/> > </disk> > <disk type='file' device='disk'> > <driver name='qemu' type='qcow2'/> > <source file='/var/lib/libvirt/images/stg6.qcow2'/> > <backingStore/> > <target dev='vdf' bus='virtio'/> > <address type='pci' domain='0x0000' bus='0x08' slot='0x00' > function='0x3' multifunction='on'/> > </disk> > <disk type='file' device='disk'> > <driver name='qemu' type='qcow2'/> > <source file='/var/lib/libvirt/images/stg7.qcow2'/> > <backingStore/> > <target dev='vdh' bus='virtio'/> > <address type='pci' domain='0x0000' bus='0x07' slot='0x00' > function='0x4'/> > </disk> > > [root@jgao-test1 ~]# virsh start demo > Domain demo started > > [root@jgao-test1 ~]# virsh domblklist demo > Target Source > ---------------------------------------------------------------- > vda /var/lib/libvirt/images/RHEL-8.3-x86_64-latest.qcow2 > vdc /var/lib/libvirt/images/stg2.qcow2 > vdd /var/lib/libvirt/images/stg3.qcow2 > vdf /var/lib/libvirt/images/stg6.qcow2 > vdh /var/lib/libvirt/images/stg7.qcow2 > > [root@jgao-test1 ~]# virsh detach-disk demo vdf > error: No disk found whose source path or target is vdf ------>> > detach failed but all disks all losed > > [root@jgao-test1 ~]# virsh domblklist demo > Target Source > ---------------------------------------------------------------- > vda /var/lib/libvirt/images/RHEL-8.3-x86_64-latest.qcow2 > > Are these expected result for you ? I think the steps above cannot match the origin bug report. The origin bug report(no additional info) did as following steps: 1. Start an VM, pc machine type 2. Hot-plug file backend disk. Disks xml: cache='none' io='thread'. All disks shared the same bus&slot but different function. 3. Hot-Unplug the function 0 disk. Check if the other exists. Please check it and reproduce that again.
BTW, are the parameters 'aio' and 'cache' necessary to reproduce this bug? Could you try without them?
(In reply to Han Han from comment #7) > (In reply to gaojianan from comment #5) > > Try without hotplug but coldplug: > > disk xml: > > <disk type='file' device='disk'> > > <driver name='qemu' type='qcow2'/> > > <source file='/var/lib/libvirt/images/RHEL-8.3-x86_64-latest.qcow2'/> > > <backingStore/> > > <target dev='vda' bus='virtio'/> > > <address type='pci' domain='0x0000' bus='0x01' slot='0x00' > > function='0x0'/> > > </disk> > > <disk type='file' device='disk'> > > <driver name='qemu' type='qcow2'/> > > <source file='/var/lib/libvirt/images/stg2.qcow2'/> > > <backingStore/> > > <target dev='vdc' bus='virtio'/> > > <address type='pci' domain='0x0000' bus='0x04' slot='0x00' > > function='0x1'/> > > </disk> > > <disk type='file' device='disk'> > > <driver name='qemu' type='qcow2'/> > > <source file='/var/lib/libvirt/images/stg3.qcow2'/> > > <backingStore/> > > <target dev='vdd' bus='virtio'/> > > <address type='pci' domain='0x0000' bus='0x06' slot='0x00' > > function='0x2'/> > > </disk> > > <disk type='file' device='disk'> > > <driver name='qemu' type='qcow2'/> > > <source file='/var/lib/libvirt/images/stg6.qcow2'/> > > <backingStore/> > > <target dev='vdf' bus='virtio'/> > > <address type='pci' domain='0x0000' bus='0x08' slot='0x00' > > function='0x3' multifunction='on'/> > > </disk> > > <disk type='file' device='disk'> > > <driver name='qemu' type='qcow2'/> > > <source file='/var/lib/libvirt/images/stg7.qcow2'/> > > <backingStore/> > > <target dev='vdh' bus='virtio'/> > > <address type='pci' domain='0x0000' bus='0x07' slot='0x00' > > function='0x4'/> > > </disk> > > > > [root@jgao-test1 ~]# virsh start demo > > Domain demo started > > > > [root@jgao-test1 ~]# virsh domblklist demo > > Target Source > > ---------------------------------------------------------------- > > vda /var/lib/libvirt/images/RHEL-8.3-x86_64-latest.qcow2 > > vdc /var/lib/libvirt/images/stg2.qcow2 > > vdd /var/lib/libvirt/images/stg3.qcow2 > > vdf /var/lib/libvirt/images/stg6.qcow2 > > vdh /var/lib/libvirt/images/stg7.qcow2 > > > > [root@jgao-test1 ~]# virsh detach-disk demo vdf > > error: No disk found whose source path or target is vdf ------>> > > detach failed but all disks all losed > > > > [root@jgao-test1 ~]# virsh domblklist demo > > Target Source > > ---------------------------------------------------------------- > > vda /var/lib/libvirt/images/RHEL-8.3-x86_64-latest.qcow2 > > > > Are these expected result for you ? > > I think the steps above cannot match the origin bug report. > The origin bug report(no additional info) did as following steps: > 1. Start an VM, pc machine type > 2. Hot-plug file backend disk. Disks xml: cache='none' io='thread'. All > disks shared the same bus&slot but different function. > 3. Hot-Unplug the function 0 disk. Check if the other exists. > > Please check it and reproduce that again. I don't think it's necessary to add cache='none' io='thread' for this bug. and as https://bugzilla.redhat.com/show_bug.cgi?id=1850412#c4,libvirt dosen't support to add disks with different functions with the same slot. So i only test with coldplug.
Using static disk in guest then hotunplug get same result: And it is irrelevant with cache and aio. 1.Boot vm with multi-function disks. /usr/libexec/qemu-kvm \ -name 'avocado-vt-vm1' \ -sandbox on \ -machine q35 \ -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \ -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 \ -nodefaults \ -device VGA,bus=pcie.0,addr=0x2 \ -m 7168 \ -smp 4,maxcpus=4,cores=2,threads=1,dies=1,sockets=2 \ -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \ -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -blockdev node-name=file_image1,driver=file,aio=threads,filename=/home/kvm_autotest_root/images/rhel830-64-virtio.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_image1,driver=qcow2,cache.direct=on,cache.no-flush=off,file=file_image1 \ -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \ -device virtio-blk-pci,id=image1,drive=drive_image1,bootindex=0,write-cache=on,multifunction=on,bus=pcie-root-port-2,addr=0x0 \ -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \ -device virtio-net-pci,mac=9a:f4:f0:0e:dc:05,id=id4dZ4px,netdev=idnbbqEr,bus=pcie-root-port-3,addr=0x0 \ -netdev tap,id=idnbbqEr,vhost=on \ -vnc :5 \ -rtc base=localtime,clock=host,driftfix=slew \ -boot menu=off,order=cdn,once=c,strict=off \ -enable-kvm -monitor stdio \ -qmp tcp:0:5955,server,nowait \ -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,slot=4,addr=0x4,chassis=4 \ -blockdev node-name=file_stg1,driver=file,filename=/home/kvm_autotest_root/images/stg1.qcow2 \ -blockdev node-name=drive_stg1,driver=qcow2,file=file_stg1 \ -blockdev node-name=file_stg2,driver=file,filename=/home/kvm_autotest_root/images/stg2.qcow2 \ -blockdev node-name=drive_stg2,driver=qcow2,file=file_stg2 \ -blockdev node-name=file_stg3,driver=file,filename=/home/kvm_autotest_root/images/stg3.qcow2 \ -blockdev node-name=drive_stg3,driver=qcow2,file=file_stg3 \ -blockdev node-name=file_stg4,driver=file,filename=/home/kvm_autotest_root/images/stg4.qcow2 \ -blockdev node-name=drive_stg4,driver=qcow2,file=file_stg4 \ -blockdev node-name=file_stg5,driver=file,filename=/home/kvm_autotest_root/images/stg5.qcow2 \ -blockdev node-name=drive_stg5,driver=qcow2,file=file_stg5 \ -blockdev node-name=file_stg6,driver=file,filename=/home/kvm_autotest_root/images/stg6.qcow2 \ -blockdev node-name=drive_stg6,driver=qcow2,file=file_stg6 \ -blockdev node-name=file_stg7,driver=file,filename=/home/kvm_autotest_root/images/stg7.qcow2 \ -blockdev node-name=drive_stg7,driver=qcow2,file=file_stg7 \ -blockdev node-name=file_stg8,driver=file,filename=/home/kvm_autotest_root/images/stg8.qcow2 \ -blockdev node-name=drive_stg8,driver=qcow2,file=file_stg8 \ \ -device virtio-blk-pci,id=stg1,drive=drive_stg1,multifunction=on,bus=pcie_extra_root_port_0,addr=0x0.1 \ -device virtio-blk-pci,id=stg2,drive=drive_stg2,multifunction=on,bus=pcie_extra_root_port_0,addr=0x0.2 \ -device virtio-blk-pci,id=stg3,drive=drive_stg3,multifunction=on,bus=pcie_extra_root_port_0,addr=0x0.3 \ -device virtio-blk-pci,id=stg4,drive=drive_stg4,multifunction=on,bus=pcie_extra_root_port_0,addr=0x0.4 \ -device virtio-blk-pci,id=stg5,drive=drive_stg5,multifunction=on,bus=pcie_extra_root_port_0,addr=0x0.5 \ -device virtio-blk-pci,id=stg6,drive=drive_stg6,multifunction=on,bus=pcie_extra_root_port_0,addr=0x0.6 \ -device virtio-blk-pci,id=stg7,drive=drive_stg7,multifunction=on,bus=pcie_extra_root_port_0,addr=0x0.7 \ -device virtio-blk-pci,id=stg8,drive=drive_stg8,multifunction=on,bus=pcie_extra_root_port_0,addr=0x0.0 \ 2.hotunplug non-zero function disk {"execute":"qmp_capabilities"} {"return": {}} {"execute":"device_del","arguments":{"id":"stg6"}} {"return": {}} {"timestamp": {"seconds": 1593423426, "microseconds": 588189}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral/stg8/virtio-backend"}} {"timestamp": {"seconds": 1593423426, "microseconds": 601051}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral/stg1/virtio-backend"}} {"timestamp": {"seconds": 1593423426, "microseconds": 613530}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral/stg2/virtio-backend"}} {"timestamp": {"seconds": 1593423426, "microseconds": 627051}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral/stg3/virtio-backend"}} {"timestamp": {"seconds": 1593423426, "microseconds": 640088}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral/stg4/virtio-backend"}} {"timestamp": {"seconds": 1593423426, "microseconds": 652629}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral/stg5/virtio-backend"}} {"timestamp": {"seconds": 1593423426, "microseconds": 664751}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral/stg6/virtio-backend"}} {"timestamp": {"seconds": 1593423426, "microseconds": 676237}, "event": "DEVICE_DELETED", "data": {"path": "/machine/peripheral/stg7/virtio-backend"}} {"timestamp": {"seconds": 1593423426, "microseconds": 684594}, "event": "DEVICE_DELETED", "data": {"device": "stg8", "path": "/machine/peripheral/stg8"}} {"timestamp": {"seconds": 1593423426, "microseconds": 686018}, "event": "DEVICE_DELETED", "data": {"device": "stg1", "path": "/machine/peripheral/stg1"}} {"timestamp": {"seconds": 1593423426, "microseconds": 687223}, "event": "DEVICE_DELETED", "data": {"device": "stg2", "path": "/machine/peripheral/stg2"}} {"timestamp": {"seconds": 1593423426, "microseconds": 688304}, "event": "DEVICE_DELETED", "data": {"device": "stg3", "path": "/machine/peripheral/stg3"}} {"timestamp": {"seconds": 1593423426, "microseconds": 689402}, "event": "DEVICE_DELETED", "data": {"device": "stg4", "path": "/machine/peripheral/stg4"}} {"timestamp": {"seconds": 1593423426, "microseconds": 690384}, "event": "DEVICE_DELETED", "data": {"device": "stg5", "path": "/machine/peripheral/stg5"}} {"timestamp": {"seconds": 1593423426, "microseconds": 691432}, "event": "DEVICE_DELETED", "data": {"device": "stg6", "path": "/machine/peripheral/stg6"}} {"timestamp": {"seconds": 1593423426, "microseconds": 692411}, "event": "DEVICE_DELETED", "data": {"device": "stg7", "path": "/machine/peripheral/stg7"}} All disks in same bus were deleted.
First of all let me change the subcomponent to PCI because this is not a storage question. I suppose you'll get the same result with any other PCI device. You always need to keep in mind that a multifunction device is just that: One device with multiple functions. It's not multiple devices, but one virtual PCI card. You can unplug it only as a whole, you can't rip out individual functions from it. If device_del were a new interface to be designed now, I could see an argument for making device_del work only on function 0 and return an error if you try it on any other function. But it's an existing interface and preserving compatibility probably means that we can't change it (and as long as QMP clients understand what it means, it's not a problem either). So I think someone who regularly works on PCI should have a look at this, but I strongly suspect this is NOTABUG.
Concur. You cannot plug PCI *functions*, only PCI *devices*. The way QEMU models PCI is plain weird in places. Multifunction devices is one of these places. To plug a multifunction device, you have to pretent you're plugging each of the functions separately as if it was a separate device, with -device / device_add. This doesn't actually plug anything until you plug in function 0. The all the functions "plugged in" so far get plugged in together as a device. From then on, you cannot plug in additional functions. To unplug a multifunction device, you unplug any of its functions. I'd recommend to always unplug function 0, just because this is slightly less insane. As far as I can tell, the reporter expects unplug of a function to unplug just the function, but observes it unplugs the device. Works as intended, NOTABUG. Actually closing the bug left to the PCI guys, just in case I'm deluded about PCI in QEMU :)
Based on the above, due to the age of the bug reassigning to Amnon's team for further triage
@Markus Armbruster I verified that your comment is exactly what is happening. From PCI point of view nothing can be done - the spec doesn't allow hotplug of functions. We indeed only inform the guest of hotplug when function 0 appears, and we 'push' the attention button that tells the guest to remove the whole pcie device when any of the functions are hot-unplugged. I think that to make it a bit more user friendly we can make qemu throw an error if user attempts to hot-unplug non-zero function with a nice error message. But that can break backward compatibility with libvirt so I'll let libvirt folks decide if this is worth it. From my point of view it is basically matter of adding 1-2 lines to qemu's pcie_cap_slot_unplug_request_cb Best regards, Maxim Levitsky PS: I am adding few libvirt developers to hear your opinion
Everything that's been said about "don't do that", "that's not how this works" etc are all correct. But we don't need to rely on engineers in bug reports warning us about this - according to the libvirt source, and my test just now both with two network interfaces and two virtio-pci disk devices plugged into different functions of the same slot, libvirt fails any attempt to hot unplug any guest device that is on the same slot as another device (so it not only fails attempts to unplug the device in function 1, but also the device in function 0). The error message it logs is this: cannot hot unplug disk device with multifunction PCI guest address 0000:00:03.1 In current libvirt the code that makes this check is common for all PCI devices. This check was done separately for each type of device prior to a refactoring of the code in libvirt-5.2.0, so the exact error message would be slightly different for older versions, but the checks to prevent hot unplugging multifunction PCI devices has been in libvirt since libvirt-0.9.3, which was released in June 2011! As for the test described in Comment 5 - it suffers a flaw that renders its results useless: the address of each disk has a different bus number, so they are not multiple functions on a single PCI slot/device, but are are each on their own slot/device. I would be mildly surprised if the guest OS actually detected all those disks in the first place, since there is no requirement that an OS probe non-0 functions for a slot that has nothing in function 0. So in the end, 1) nobody will run qemu directly, so the error in the original BZ description can't be encountered by any user who is using qemu in the documented supported manner. And 2) the test using libvirt was constructed incorrectly, so it isn't providing the requested results, but 3) when libvirt is provided a proper example of a multifunction device, attempts to hot unplug one of the functions will properly result in a failure of the request and an error log. I agree that this should be closed as NOTABUG (or if you want to be really pedantic, as WONTFIX, since it is true that qemu doesn't behave in a friendly matter. But we've already decided that we don't care about that, because libvirt is protecting the user from the ungriendly behavior). Also, this is inconsequential, but in Comment 5 I noticed this: > <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x3' multifunction='on'/> The "multifunction" flag should normally be set only for the device at function='0'. When we first implemented that feature years ago, someone noticed that one piece of actual hardware (it may have been my 2009 model IBM Thinkstation) showed the multifunction bit set for all functions of a slot with multiple functions, but in all other cases only function 0 had the multifunction flag set. Additionally, libvirt will automatically set multifunction='on' for function 0 when it sees that multiple functions are in use, so you should never need to explicitly set it at all (it does seem to be harmless if you set it for non-0 functions though)
Oh, and I should answer Maxim's question about backward compatibility (and add a bit more info that I had forgotten in my previous comment)... As far as *backward* compatibility, since we explicitly prohibit hot unplug of any function of a multifunction device, and don't even have a method of hot unplugging multiple functions at the same time, there are no backward compatibility concerns wrt libvirt. However, there is a set of patches that's been lingering around for a long time that *would* permit hot unplugging all functions of a multifunction device. From the libvirt POV, the user provides XML for all the functions that they want unplugged. Once libvirt has verified that all the functions of the device are in the request, it either a) issues a device_del command for function 0 of the device, which will cause all functions to be unplugged, or b) on Power PC hosts running qemu < 4.2, it issues a device_del for each non-0 function, then a separate device_del for function 0 (according to the patches, the behavior for Power PC was changed in qemu 4.2 to behave more like x86). Since there are apparently people who would like to be able to unplug multifunction devices, I don't think qemu should disable this capability.
Closing as Notabug based on comment#23 Feel free to reopen if you see it differently
@Laine Stump Thanks for the info. Since libvirt doesn't allow hot-unplug of multi-function device at all, I do think that it would be worth it to add error to qemu when unplug of non zero function is attempted, and maybe even some nice message about 'about to unplug all functions' when unplug of function 0 is done. Qemu direct use does happen, even if it is only by us / QE, and the more user friendly qemu is, the better is for all of us. I'll prepare a patch for this eventually, as upstream only low priority work.
(In reply to Maxim Levitsky from comment #26)> > I do > think that it would be worth it to add error to qemu when unplug of non zero > function is attempted, > and maybe even some nice message about 'about to unplug all functions' when > unplug of function 0 is done. Anything that's done to qemu should be careful not to disrupt the assumption (made at least by the person who wrote the so-far-unpushed libvirt multifunction hotplug patches) that sending device_del for function 0 of a multifunction device should detach all functions of the device, with no extra complaining. I'm unsure about non-0 functions - as I said before, previous versions of qemu for powerpc required non-0 functions to have been issued a device_del command prior to sending device_del for function 0 (which is mimicking the way that hotplug works). I guess since qemu 4.2+ on powerpc behaves the same as x86 (device_del for function by itself will cause all functions to be detached) there wouldn't be a problem with having qemu issue an error if someone attempted device_del of function != 0.
>Anything that's done to qemu should be careful not to disrupt the assumption (made at least by the person who wrote the so-far-unpushed libvirt multifunction hotplug patches) that >sending device_del for function 0 of a multifunction device should detach all functions of the device, with no extra complaining. >I'm unsure about non-0 functions - as I said before, previous versions of qemu for powerpc required non-0 functions to have been issued a device_del command prior to sending device_del >for function 0 (which is mimicking the way that hotplug works). I guess since qemu 4.2+ on powerpc behaves the same as x86 (device_del for function by itself will cause all functions to >be detached) there wouldn't be a problem with having qemu issue an error if someone attempted device_del of function != 0. I guess we are in agreement then. I'll send a patch for that when I have some spare time, and let see if upstream likes it.