Description of problem: Hotplug hostdev type interface from interface pool will fail as "Attempted double use of PCI Address" Version-Release number of selected component (if applicable): libvirt-5.6.0-2.module+el8.1.0+4015+63576633.x86_64 How reproducible: 100% Steps to Reproduce: 1. Define and start a hostdev type network as below: # virsh net-dumpxml passthrough1 <network> <name>passthrough1</name> <uuid>7a5ff548-75f6-45bd-8564-d944a00d1321</uuid> <forward mode='hostdev' managed='yes'> <driver name='vfio'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x3'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x1'/> </forward> </network> 2. Start a vm, and hotplug an interface from above interface pool, it will fail as below: # virsh attach-interface rhel network passthrough1 error: Failed to attach interface error: XML error: Attempted double use of PCI Address 0000:09:01.0 # grep error /var/log/libvirt/libvirtd.log 2019-08-22 10:22:29.145+0000: 20376: error : virDomainPCIAddressReserveAddrInternal:854 : XML error: Attempted double use of PCI Address 0000:09:01.0 3. check the vm xml: # virsh dumpxml rhel | grep "<address type='pci' domain='0x0000' bus='0x09'" ==> no output # virsh dumpxml rhel | grep "index='9'" <controller type='pci' index='9' model='pcie-to-pci-bridge'> Actual results: Hotplug interface fail Expected results: Hotplug should not fail Additional info: 1. if add "--model virtio", it will try to attach to pcie slot and report error as well: # virsh attach-interface rhel network passthrough1 --model virtio error: Failed to attach interface error: XML error: Attempted double use of PCI Address 0000:01:00.0 2. hotplug hostdev type interface succeed and plugged into pcie slot automatically. # virsh attach-interface rhel hostdev 0000:03:10:3 --managed Interface attached successfully # virsh attach-interface rhel hostdev 0000:03:10:1 --managed --model rtl8139 Interface attached successfully # virsh dumpxml rhel | grep /interface -B8 # virsh dumpxml rhel | grep /interface -B9 </controller> <interface type='hostdev' managed='yes'> <mac address='52:54:00:c2:29:20'/> <driver name='vfio'/> <source> <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x3'/> </source> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> <interface type='hostdev' managed='yes'> <mac address='52:54:00:e9:2b:36'/> <driver name='vfio'/> <source> <address type='pci' domain='0x0000' bus='0x03' slot='0x10' function='0x1'/> </source> <model type='rtl8139'/> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/> </interface> # virsh dumpxml rhel | egrep "index='8'|index='1'" <controller type='pci' index='1' model='pcie-root-port'> <controller type='pci' index='8' model='pcie-root-port'>
I bisected upstream and hit this as the cause commit 01ca4010d86584df502458d66f4a3956cc91ce37 Author: Wang Yechao <wang.yechao255.cn> Date: Mon Dec 17 19:30:34 2018 +0800 qemu: Assign device addresses earlier in qemuDomainAttachNetDevice If code in the @actualType switch needs to have/know which PCI Address is being used, then we must assign it earlier. In particular a vhost-user device needs to call qemuDomainSupportsNicdev which requires an address to be defined. Signed-off-by: Wang Yechao <wang.yechao255.cn> Reviewed-by: John Ferlan <jferlan> IIUC, what's happening is that qemuDomainAttachNetDevice reserves the PCI address. Since this is a hostdev NIC, we then call qemuDmoainAttachHostDevice, which also tries to reserve the PCI address, and thus fails.
The bug first appeared in libvirt v5.1.0 Posted a patch that fixes this upstream: https://www.redhat.com/archives/libvir-list/2019-October/msg01291.html
Pushed upstream. Will be in upstream libvirt 5.10: commit 47a7b8a96b6343d4af18ef80330f805ef031fe9b (HEAD -> master, origin/master, origin/HEAD) Author: Laine Stump <laine> Date: Sat Oct 19 01:58:07 2019 -0400 qemu: avoid double reservation of PCI address for interface type='hostdev'
Reproduce this bug on libvirt-5.6.0-7.module+el8.1.1+4483+2f45aaa2.x86_64. Version: libvirt-5.6.0-7.module+el8.1.1+4483+2f45aaa2.x86_64 qemu-kvm-4.1.0-15.module+el8.1.1+4700+209eec8f.x86_64 kernel-4.18.0-147.0.3.el8_1.x86_64 Steps: # virsh net-list Name State Autostart Persistent ------------------------------------------------- passthrough1 active no yes # virsh net-dumpxml passthrough1 <network> <name>passthrough1</name> <uuid>7a5ff548-75f6-45bd-8564-d944a00d1321</uuid> <forward mode='hostdev' managed='yes'> <driver name='vfio'/> <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x1'/> <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x3'/> <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x5'/> <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x7'/> <address type='pci' domain='0x0000' bus='0x82' slot='0x11' function='0x1'/> <address type='pci' domain='0x0000' bus='0x82' slot='0x11' function='0x3'/> <address type='pci' domain='0x0000' bus='0x82' slot='0x11' function='0x5'/> </forward> </network> # virsh domstate test shut off # virsh start test Domain test started # virsh attach-interface test network passthrough1 error: Failed to attach interface error: XML error: Attempted double use of PCI Address 0000:02:03.0 # virsh attach-interface test network passthrough1 --model virtio error: Failed to attach interface error: XML error: Attempted double use of PCI Address 0000:05:00.0 Verify this bug on libvirt-5.6.0-8.module+el8.1.1+4828+0f2d78eb.x86_64. Steps: # yum update libvirt* -y # systemctl restart libvirtd # rpm -qa libvirt libvirt-5.6.0-8.module+el8.1.1+4828+0f2d78eb.x86_64 # virsh attach-interface test network passthrough1 Interface attached successfully # virsh dumpxml test |grep "hostdev" -A7 <interface type='hostdev' managed='yes'> <mac address='52:54:00:34:9a:93'/> <driver name='vfio'/> <source> <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x1'/> </source> <model type='rtl8139'/> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </interface> # virsh detach-interface --domain test --mac 52:54:00:34:9a:93 --type hostdev Interface detached successfully # virsh dumpxml test |grep "hostdev" -A7 No output # virsh attach-interface test network passthrough1 --model virtio Interface attached successfully # virsh dumpxml test |grep "hostdev" -A7 <interface type='hostdev' managed='yes'> <mac address='52:54:00:40:c1:a8'/> <driver name='vfio'/> <source> <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x3'/> </source> <model type='virtio'/> <alias name='hostdev0'/> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </interface>
During verifying this bug, I found another problem which is I can continue to define net with same name and uuid for many times. Could you please whether this is expected and should I file a new bug to track it? Version: qemu-kvm-4.1.0-15.module+el8.1.1+4700+209eec8f.x86_64 libvirt-5.6.0-8.module+el8.1.1+4828+0f2d78eb.x86_64 kernel-4.18.0-147.0.3.el8_1.x86_64 Steps: 1. There is no other net here and prepare two xml with same name and uuid. # virsh net-list --all Name State Autostart Persistent -------------------------------------------- default active yes yes # cat net.xml <network> <name>passthrough1</name> <uuid>7a5ff548-75f6-45bd-8564-d944a00d1321</uuid> <forward mode='hostdev' managed='yes'> <driver name='vfio'/> <address type="pci" domain='0x0000' bus='0x82' slot='0x10' function='0x1'/> <address type="pci" domain='0x0000' bus='0x82' slot='0x10' function='0x3'/> <address type="pci" domain='0x0000' bus='0x82' slot='0x10' function='0x5'/> <address type="pci" domain='0x0000' bus='0x82' slot='0x10' function='0x7'/> <address type="pci" domain='0x0000' bus='0x82' slot='0x11' function='0x1'/> <address type="pci" domain='0x0000' bus='0x82' slot='0x11' function='0x3'/> <address type="pci" domain='0x0000' bus='0x82' slot='0x11' function='0x5'/> </forward> </network> # cat net1.xml <network> <name>passthrough1</name> <uuid>7a5ff548-75f6-45bd-8564-d944a00d1321</uuid> <forward mode='hostdev' managed='yes'> <driver name='vfio'/> <address type="pci" domain='0x0000' bus='0x82' slot='0x10' function='0x1'/> </forward> </network> 2. Define net from the first xml file # virsh net-define net1.xml Network passthrough1 defined from net1.xml # virsh net-start passthrough1 Network passthrough1 started # virsh net-list --all Name State Autostart Persistent ------------------------------------------------- default active yes yes passthrough1 active no yes # virsh net-dumpxml passthrough1 <network> <name>passthrough1</name> <uuid>7a5ff548-75f6-45bd-8564-d944a00d1321</uuid> <forward mode='hostdev' managed='yes'> <driver name='vfio'/> <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x1'/> <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x3'/> <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x5'/> <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x7'/> <address type='pci' domain='0x0000' bus='0x82' slot='0x11' function='0x1'/> <address type='pci' domain='0x0000' bus='0x82' slot='0x11' function='0x3'/> <address type='pci' domain='0x0000' bus='0x82' slot='0x11' function='0x5'/> </forward> </network> 3. Define net from the second xml file # virsh net-define net1.xml Network passthrough1 defined from net1.xml # virsh net-start passthrough1 error: Failed to start network passthrough1 error: Requested operation is not valid: network is already active # virsh net-destroy passthrough1 Network passthrough1 destroyed # virsh net-start passthrough1 Network passthrough1 started # virsh net-list --all Name State Autostart Persistent ------------------------------------------------- default active yes yes passthrough1 active no yes # virsh net-dumpxml passthrough1 (No need to undefine the firt definition, and i can see the following result.) <network> <name>passthrough1</name> <uuid>7a5ff548-75f6-45bd-8564-d944a00d1321</uuid> <forward mode='hostdev' managed='yes'> <driver name='vfio'/> <address type='pci' domain='0x0000' bus='0x82' slot='0x10' function='0x1'/> </forward> </network>
(In reply to jiyan from comment #8) > During verifying this bug, I found another problem which is I can continue > to define net with same name and uuid for many times. > Could you please whether this is expected and should I file a new bug to > track it? That is proper behavior. You're not "definite a new network", you are "re-defining the same network". This is the documented way to modify the network definition. (if you were able to define another network with the same name and different uuid, or different name and same uuid, then *that* would be a bug)
s/definite/defining/ sigh.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0404