Description of problem: libvirt defined guest xml pci slot number is different with the pci number of the guest lspci shows Version-Release number of selected component (if applicable): [root@dell-per740-43 ~]# uname -a Linux dell-per740-43.rhts.eng.pek2.redhat.com 4.18.0-348.el8.x86_64 #1 SMP Mon Oct 4 12:17:22 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux [root@dell-per740-43 ~]# lspci -s 5e:00.0 -v 5e:00.0 Ethernet controller: Mellanox Technologies MT2892 Family [ConnectX-6 Dx] Subsystem: Mellanox Technologies Device 0083 Flags: bus master, fast devsel, latency 0, IRQ 206, NUMA node 0, IOMMU group 80 Memory at bc000000 (64-bit, prefetchable) [size=32M] Expansion ROM at b8800000 [disabled] [size=1M] Capabilities: [60] Express Endpoint, MSI 00 Capabilities: [48] Vital Product Data Capabilities: [9c] MSI-X: Enable+ Count=64 Masked- Capabilities: [c0] Vendor Specific Information: Len=18 <?> Capabilities: [40] Power Management version 3 Capabilities: [100] Advanced Error Reporting Capabilities: [150] Alternative Routing-ID Interpretation (ARI) Capabilities: [180] Single Root I/O Virtualization (SR-IOV) Capabilities: [1c0] Secondary PCI Express Capabilities: [230] Access Control Services Capabilities: [320] Lane Margining at the Receiver <?> Capabilities: [370] Physical Layer 16.0 GT/s <?> Capabilities: [420] Data Link Feature <?> Kernel driver in use: mlx5_core Kernel modules: mlx5_core [root@dell-per740-43 ~]# rpm -qa | grep libvirt libvirt-daemon-driver-storage-iscsi-direct-7.6.0-6.module+el8.5.0+13051+7ddbe958.x86_64 libvirt-daemon-config-network-7.6.0-6.module+el8.5.0+13051+7ddbe958.x86_64 libvirt-glib-3.0.0-1.el8.x86_64 libvirt-daemon-driver-storage-core-7.6.0-6.module+el8.5.0+13051+7ddbe958.x86_64 libvirt-daemon-driver-storage-rbd-7.6.0-6.module+el8.5.0+13051+7ddbe958.x86_64 libvirt-client-7.6.0-6.module+el8.5.0+13051+7ddbe958.x86_64 libvirt-daemon-7.6.0-6.module+el8.5.0+13051+7ddbe958.x86_64 libvirt-daemon-driver-storage-disk-7.6.0-6.module+el8.5.0+13051+7ddbe958.x86_64 libvirt-daemon-driver-storage-mpath-7.6.0-6.module+el8.5.0+13051+7ddbe958.x86_64 libvirt-7.6.0-6.module+el8.5.0+13051+7ddbe958.x86_64 libvirt-libs-7.6.0-6.module+el8.5.0+13051+7ddbe958.x86_64 libvirt-daemon-config-nwfilter-7.6.0-6.module+el8.5.0+13051+7ddbe958.x86_64 libvirt-daemon-driver-storage-logical-7.6.0-6.module+el8.5.0+13051+7ddbe958.x86_64 libvirt-daemon-driver-secret-7.6.0-6.module+el8.5.0+13051+7ddbe958.x86_64 libvirt-daemon-driver-nodedev-7.6.0-6.module+el8.5.0+13051+7ddbe958.x86_64 libvirt-daemon-driver-storage-7.6.0-6.module+el8.5.0+13051+7ddbe958.x86_64 libvirt-daemon-driver-qemu-7.6.0-6.module+el8.5.0+13051+7ddbe958.x86_64 python3-libvirt-7.6.0-1.module+el8.5.0+12098+85b3670b.x86_64 libvirt-daemon-kvm-7.6.0-6.module+el8.5.0+13051+7ddbe958.x86_64 libvirt-daemon-driver-nwfilter-7.6.0-6.module+el8.5.0+13051+7ddbe958.x86_64 libvirt-daemon-driver-storage-scsi-7.6.0-6.module+el8.5.0+13051+7ddbe958.x86_64 libvirt-daemon-driver-interface-7.6.0-6.module+el8.5.0+13051+7ddbe958.x86_64 libvirt-daemon-driver-storage-gluster-7.6.0-6.module+el8.5.0+13051+7ddbe958.x86_64 libvirt-daemon-driver-storage-iscsi-7.6.0-6.module+el8.5.0+13051+7ddbe958.x86_64 libvirt-daemon-driver-network-7.6.0-6.module+el8.5.0+13051+7ddbe958.x86_64 libvirt-devel-7.6.0-6.module+el8.5.0+13051+7ddbe958.x86_64 [root@dell-per740-43 ~]# [root@dell-per740-43 ~]# rpm -qa | grep qemu qemu-kvm-block-ssh-6.0.0-33.module+el8.5.0+13041+05be2dc6.x86_64 qemu-kvm-ui-opengl-6.0.0-33.module+el8.5.0+13041+05be2dc6.x86_64 qemu-kvm-ui-spice-6.0.0-33.module+el8.5.0+13041+05be2dc6.x86_64 qemu-kvm-docs-6.0.0-33.module+el8.5.0+13041+05be2dc6.x86_64 qemu-kvm-common-6.0.0-33.module+el8.5.0+13041+05be2dc6.x86_64 qemu-kvm-block-iscsi-6.0.0-33.module+el8.5.0+13041+05be2dc6.x86_64 ipxe-roms-qemu-20181214-8.git133f4c47.el8.noarch qemu-kvm-block-gluster-6.0.0-33.module+el8.5.0+13041+05be2dc6.x86_64 qemu-kvm-hw-usbredir-6.0.0-33.module+el8.5.0+13041+05be2dc6.x86_64 libvirt-daemon-driver-qemu-7.6.0-6.module+el8.5.0+13051+7ddbe958.x86_64 qemu-kvm-block-curl-6.0.0-33.module+el8.5.0+13041+05be2dc6.x86_64 qemu-kvm-6.0.0-33.module+el8.5.0+13041+05be2dc6.x86_64 qemu-img-6.0.0-33.module+el8.5.0+13041+05be2dc6.x86_64 qemu-kvm-block-rbd-6.0.0-33.module+el8.5.0+13041+05be2dc6.x86_64 qemu-kvm-core-6.0.0-33.module+el8.5.0+13041+05be2dc6.x86_64 How reproducible: 100% Steps to Reproduce: Create two vdpa_vhost interfaces Fill vdpa_vhost interfaces into guest virsh start guest virsh dumpxml guest exec lpci inside guest Compare the pci slots between xml output and inside guest lspci show . :: [ 04:21:33 ] :: [ BEGIN ] :: Running 'modprobe -r mlx5_vdpa' :: [ 04:21:33 ] :: [ BEGIN ] :: Running 'modprobe -r virtio_vdpa' :: [ 04:21:34 ] :: [ BEGIN ] :: Running 'modprobe -r vhost_vdpa' :: [ 04:21:34 ] :: [ BEGIN ] :: Running 'modprobe -r vdpa' :: [ 04:21:34 ] :: [ BEGIN ] :: Running 'modprobe vdpa' :: [ 04:21:34 ] :: [ BEGIN ] :: Running 'modprobe vhost_vdpa' :: [ 04:21:35 ] :: [ BEGIN ] :: Running 'modprobe mlx5_vdpa' :: [ 04:21:36 ] :: [ BEGIN ] :: Running 'ip li set ens3f0 up' :: [ 04:21:45 ] :: [ BEGIN ] :: Running 'ip li set ens3f0 vf 0 spoofchk off trust on' :: [ 04:21:47 ] :: [ BEGIN ] :: Running 'ip li set ens3f1 up' :: [ 04:21:56 ] :: [ BEGIN ] :: Running 'ip li set ens3f1 vf 0 spoofchk off trust on' :: [ 04:21:58 ] :: [ BEGIN ] :: Running 'ulimit -l unlimited' :: [ 04:21:59 ] :: [ BEGIN ] :: Running 'nmcli device set ens3f0 managed no' :: [ 04:21:59 ] :: [ BEGIN ] :: Running 'nmcli device set ens3f1 managed no' :: [ 04:21:59 ] :: [ BEGIN ] :: Running 'echo 0000:5e:00.2 >/sys/bus/pci/drivers/mlx5_core/unbind' :: [ 04:22:01 ] :: [ BEGIN ] :: Running 'echo 0000:5e:01.2 >/sys/bus/pci/drivers/mlx5_core/unbind' :: [ 04:22:02 ] :: [ BEGIN ] :: Running 'sleep 2' :: [ 04:22:04 ] :: [ BEGIN ] :: Running 'devlink dev eswitch set pci/0000:5e:00.0 mode switchdev' :: [ 04:22:06 ] :: [ BEGIN ] :: Running 'devlink dev eswitch set pci/0000:5e:00.1 mode switchdev' :: [ 04:22:08 ] :: [ BEGIN ] :: Running 'echo 0000:5e:00.2 >/sys/bus/pci/drivers/mlx5_core/bind' :: [ 04:22:09 ] :: [ BEGIN ] :: Running 'echo 0000:5e:01.2 >/sys/bus/pci/drivers/mlx5_core/bind' :: [ 04:22:10 ] :: [ BEGIN ] :: Running 'ethtool -K ens3f0v0 hw-tc-offload on' :: [ 04:22:10 ] :: [ BEGIN ] :: Running 'ethtool -K ens3f1v0 hw-tc-offload on' :: [ 04:22:10 ] :: [ BEGIN ] :: Running 'nmcli device set ens3f0v0 managed no' :: [ 04:22:10 ] :: [ BEGIN ] :: Running 'nmcli device set ens3f1v0 managed no' :: [ 04:22:11 ] :: [ BEGIN ] :: Running 'vdpa mgmtdev show' pci/0000:5e:00.2: supported_classes net pci/0000:5e:01.2: supported_classes net :: [ 04:22:11 ] :: [ BEGIN ] :: Running 'vdpa mgmtdev show -p' pci/0000:5e:00.2: supported_classes net pci/0000:5e:01.2: supported_classes net :: [ 04:22:11 ] :: [ BEGIN ] :: Running 'vdpa mgmtdev show -j' {"mgmtdev":{"pci/0000:5e:00.2":{"supported_classes":["net"]},"pci/0000:5e:01.2":{"supported_classes":["net"]}}} :: [ 04:22:11 ] :: [ BEGIN ] :: Running 'vdpa dev add name vdpa0 mgmtdev pci/0000:5e:00.2' :: [ 04:22:12 ] :: [ BEGIN ] :: Running 'vdpa dev add name vdpa1 mgmtdev pci/0000:5e:01.2' :: [ 04:22:12 ] :: [ BEGIN ] :: Running 'nmcli device set ens3f0 managed no' :: [ 04:22:12 ] :: [ BEGIN ] :: Running 'nmcli device set ens3f1 managed no' :: [ 04:22:13 ] :: [ BEGIN ] :: Running 'nmcli device set eth0 managed no' :: [ 04:22:13 ] :: [ BEGIN ] :: Running 'nmcli device set eth1 managed no' :: [ 04:22:13 ] :: [ BEGIN ] :: Running 'ip li set eth0 up' :: [ 04:22:14 ] :: [ BEGIN ] :: Running 'ip li set eth1 up' :: [ 04:22:14 ] :: [ BEGIN ] :: Running 'ip li show eth0' 135: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether 7a:ec:94:05:7c:2c brd ff:ff:ff:ff:ff:ff :: [ 04:22:14 ] :: [ BEGIN ] :: Running 'ip li show eth1' 136: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether de:82:ba:ae:2a:91 brd ff:ff:ff:ff:ff:ff :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: Define and start guest :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: :: [ 04:24:17 ] :: [ BEGIN ] :: Running 'virsh start gg' Domain 'gg' started :: [ 04:25:20 ] :: [ BEGIN ] :: Running 'virsh dumpxml gg' <domain type='kvm' id='9'> <name>gg</name> <uuid>4612b386-410c-11ec-b138-5405dbd35b7b</uuid> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>8388608</currentMemory> <memoryBacking> <hugepages> <page size='1048576' unit='KiB'/> </hugepages> <access mode='shared'/> </memoryBacking> <vcpu placement='static'>3</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='4'/> <vcpupin vcpu='2' cpuset='6'/> <emulatorpin cpuset='12'/> </cputune> <numatune> <memory mode='strict' nodeset='0'/> </numatune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-rhel8.5.0'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pmu state='off'/> <vmport state='off'/> <ioapic driver='qemu'/> </features> <cpu mode='host-passthrough' check='none' migratable='on'> <feature policy='require' name='tsc-deadline'/> <numa> <cell id='0' cpus='0-2' memory='8388608' unit='KiB' memAccess='shared'/> </numa> </cpu> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/var/tmp//gg.qcow2' index='1'/> <backingStore/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </disk> <controller type='usb' index='0' model='none'> <alias name='usb'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x10'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x11'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0x8'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x9'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xa'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xb'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <controller type='sata' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0x8'/> <alias name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:bb:63:7b'/> <source bridge='virbr0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </interface> <interface type='vdpa'> <mac address='52:54:00:11:8f:ea'/> <source dev='/dev/vhost-vdpa-0'/> <model type='virtio'/> <alias name='net1'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </interface> <interface type='vdpa'> <mac address='52:54:00:11:8f:eb'/> <source dev='/dev/vhost-vdpa-1'/> <model type='virtio'/> <alias name='net2'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/0'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <serial type='pty'> <source path='/dev/pts/2'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial1'/> </serial> <console type='pty' tty='/dev/pts/0'> <source path='/dev/pts/0'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-9-gg/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'> <alias name='input0'/> </input> <input type='keyboard' bus='ps2'> <alias name='input1'/> </input> <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <audio id='1' type='none'/> <video> <model type='cirrus' vram='16384' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='selinux' relabel='yes'> <label>system_u:system_r:svirt_t:s0:c474,c598</label> <imagelabel>system_u:object_r:svirt_image_t:s0:c474,c598</imagelabel> </seclabel> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+107:+1001</label> <imagelabel>+107:+1001</imagelabel> </seclabel> </domain> Login into guest [root@localhost ~]# lspci 00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller 00:01.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port 00:02.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port 00:03.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port 00:04.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port 00:05.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port 00:06.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port 00:07.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port 00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02) 00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02) 00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02) 01:00.0 Communication controller: Red Hat, Inc. Virtio console (rev 01) 02:00.0 SCSI storage controller: Red Hat, Inc. Virtio block device (rev 01) 03:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01) 04:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01) 05:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01) 06:00.0 VGA compatible controller: Cirrus Logic GD 5446 07:00.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon (rev 01) From guest xml define <interface type='bridge'> <mac address='52:54:00:bb:63:7b'/> <source bridge='virbr0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </interface> <interface type='vdpa'> <mac address='52:54:00:11:8f:ea'/> <source dev='/dev/vhost-vdpa-0'/> <model type='virtio'/> <alias name='net1'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </interface> <interface type='vdpa'> <mac address='52:54:00:11:8f:eb'/> <source dev='/dev/vhost-vdpa-1'/> <model type='virtio'/> <alias name='net2'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </interface> From inside guest 03:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01) 04:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01) 05:00.0 Ethernet controller: Red Hat, Inc. Virtio network device (rev 01) Here the xml defined pci slot number is different with the pci number of lspci shows Actual results: pci show different number in guest and xml show . Expected results: They have the same pci slot number Additional info:
Hi Jing I can reproduced this problem with qemu directly.The following points are summarized through the test： 1.A single virtio-net device can also reproduce this problem. 2.This problem can still be reproduced without adding any nic device. The storage pci slot number defined by qemu command line and the pci number displayed by lspci are also different. 3.Based on above, I guess this is PCI issue, in order to verify this, please adds relevant feature owners for testing. Thanks in advance. Qemu CLI: /usr/libexec/qemu-kvm \ -name 'avocado-vt-vm1' \ -sandbox on \ -machine q35,memory-backend=mem-machine_mem \ -device pcie-root-port,id=pcie-root-port-0,multifunction=on,bus=pcie.0,addr=0x1,chassis=1 \ -device pcie-pci-bridge,id=pcie-pci-bridge-0,addr=0x0,bus=pcie-root-port-0 \ -nodefaults \ -device VGA,bus=pcie.0,addr=0x2 \ -m 29696 \ -object memory-backend-ram,size=29696M,id=mem-machine_mem \ -smp 32,maxcpus=32,cores=16,threads=1,dies=1,sockets=2 \ -cpu 'Cascadelake-Server',ss=on,vmx=on,pdcm=on,hypervisor=on,tsc-adjust=on,umip=on,pku=on,md-clear=on,stibp=on,arch-capabilities=on,xsaves=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,rdctl-no=on,ibrs-all=on,skip-l1dfl-vmentry=on,mds-no=on,pschange-mc-no=on,tsx-ctrl=on,hle=off,rtm=off,kvm_pv_unhalt=on \ -device pcie-root-port,id=pcie-root-port-1,port=0x1,addr=0x1.0x1,bus=pcie.0,chassis=2 \ -device qemu-xhci,id=usb1,bus=pcie-root-port-1,addr=0x0 \ -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 \ -device pcie-root-port,id=pcie-root-port-2,port=0x2,addr=0x1.0x2,bus=pcie.0,chassis=3 \ -device virtio-scsi-pci,id=virtio_scsi_pci0,bus=pcie-root-port-2,addr=0x0 \ -blockdev node-name=file_image1,driver=file,auto-read-only=on,discard=unmap,aio=threads,filename=/home/kvm_autotest_root/images/rhel860-64-virtio-scsi.qcow2,cache.direct=on,cache.no-flush=off \ -blockdev node-name=drive_image1,driver=qcow2,read-only=off,cache.direct=on,cache.no-flush=off,file=file_image1 \ -device scsi-hd,id=image1,drive=drive_image1,write-cache=on \ -device pcie-root-port,id=pcie-root-port-3,port=0x3,addr=0x1.0x3,bus=pcie.0,chassis=4 \ -device virtio-net-pci,mac=9a:81:a5:83:bb:22,id=,netdev=idj2BWah,bus=pcie-root-port-3,addr=0x0 \ -netdev tap,id=idj2BWah,vhost=on \ -vnc :0 \ -rtc base=utc,clock=host,driftfix=slew \ -boot menu=off,order=cdn,once=c,strict=off \ -enable-kvm \ -device pcie-root-port,id=pcie_extra_root_port_0,multifunction=on,bus=pcie.0,addr=0x3,chassis=8 \ -monitor stdio \ Best Regrads Lei
from qemu POV it's more or less as expected https://libvirt.org/pci-addresses.html I think it's a libvirt issue it should either document this or disallow this case.
Which "case" is it that you think should be documented/disallowed? There has never been any guarantee (nor any way that we could possibly guarantee) that the bus numbers for PCI devices in the libvirt XML match the bus numbers produced by the guest kernel's enumeration of PCI devices. As a matter of fact QEMU doesn't even have a way for us to specify bus *number* at all - when a PCI controller is added on the QEMU commandline, we specify the slot.function where the controller is plugged in numerically (e,g, "addr=0x4.0x1"), but the bus that controller is plugged into is specified *by name* not number (e.g. "bus="pcie.0" - this is just an opaque string, the presence of "0" in the string has no significance whatsoever).; we also give a *name* to the new controller (e.g. "id=pci.2"). Then, when a PCI device (or another controller) is plugged into that PCI controller, we again specify which controller the new device is being plugged into *by name*, not number (again, slot.function *is* specified numerically). My understanding is that even in the PCI spec the pattern is the same - although slot and function are specified numerically (and so a match can be guaranteed), the bus number is not explicitly supplied anywhere - it is implied/derived from the topology of the tree of PCI controllers; this enumeration is done by the guest firmware/kernel, which is free to assign numbers any way it likes (I presume it usually assigns numbers in the order it scans the controllers, but it really could do whatever it wanted). So, while it is usually the case that the bus numbers in the guest just happen to match what's in the libvirt XML, this is by no means guaranteed, and there is no method available to libvirt to guarantee it. What *can* be guaranteed is that each time the guest is started with the same XML, the bus topology presented to the guest will be identical/consistent. What appears in the guest is up to the guest. I don't see a bug here, just a limitation of the PCI spec / QEMU / guest OS. Unless someone can describe how we could make the XML and guest kernel bus numbers match, this BZ should be closed as NOTABUG (or possibly CANTFIX, but that implies that there is something inherently wrong, and really the only thing wrong here is an incorrect expectation (which, admittedly, was encouraged by libvirt's XML using bus numbers instead of names (I suppose because it seemed more familiar/realistic). (NB: there is a similar issue with the target device name of disk devices - while it's actually *required* to specify this in the definition of a disk device (e.g. "<target dev='sda'/>"), there is no existing method of conveying that name down to the guest - the guest OS will name the device whatever it likes.)
Two points: 1. https://libvirt.org/pci-addresses.html makes it look like it enumerates all cases where bus numbers do not match. 2. it is not clear what does the bus number in domain xml do then. the docs say "The relationship between the PCI addresses configured in the domain XML and those seen by the guest OS can sometime seem confusing" if we now basically say there's no relationship, maybe just say so?
(In reply to Michael S. Tsirkin from comment #9) > Two points: > 1. https://libvirt.org/pci-addresses.html > makes it look like it enumerates all cases where bus numbers do not match. In my opinion, it looks like it's just providing some examples of where they don't match in order to illustrate the fact that they don't always match, and even says "that's [matching] not guaranteed to happen and will in fact not be the case in all but the simplest scenarios". Would it help to change the sentence titled "More Complex Cases" to something like: "In more complex cases, a few examples of which are below, the PCI address visible in the domain XML will correlate to the one seen by the guest OS, but the two may not match." or something like that? > 2. it is not clear what does the bus number in domain xml do then. It's just a convenient way to match up which controller a particular device is plugged into (answer: "bus" of device matches "index" of controller). I agree it's not ideal, since calling it a "bus number" can mislead the user who doesn't read pcie-addresses.html into believing that it will always match the bus number in the guest, but that's a decision that was made in 2008 or something, so it can't be changed now. > the docs say > "The relationship between the PCI addresses configured in the domain XML and > those seen by the guest OS can sometime seem confusing" (That sentence is from the place in docs/formatdomain.html that has the link to pci-addresses.html.) > if we now basically say there's no relationship, maybe just say so? There *is* a relationship, it's just not always 1:1 :-), and I think pci-addresses.html says that. How about if we add to the "sometime seem confusing" sentence in formatdomain.html, to say something like "... can sometimes seem confusing, in particular because the bus number in the libvirt config may not match the bus number reported by the guest OS running in the VM" (or something like that). Andrea, I've Cc'ed you since you were the original author, and might have an opinion...
all good suggestions. >"bus" of device matches "index" of controller is this documented anywhere? thanks
(In reply to Michael S. Tsirkin from comment #12) > >"bus" of device matches "index" of controller > is this documented anywhere? Good point! This is one of those things that's been a fact of life for so long that I've just assumed it was documented as such somewhere, but I just looked through everything about pci controllers in formatdomain.html and there is no mention of it! The only info given about the index attribute for PCI controllers is this: "pci-root controllers for pSeries guests use this attribute to record the order they will show up in the guest." which is completely useless for anybody not using PPC guests (i.e. almost everyone). The information in the section generic to all controllers (not just PCI) says this: "a mandatory attribute index which is the decimal integer describing in which order the bus controller is encountered (for use in controller attributes of <address> elements)" which is not only confusing (because it just says "controller attributes of <address> elements" rather than specifically naming the attributes) and also incorrect/misleading since it says it describes "in which order the bus controller is encountered", which is definitely wrong - PCI controllers will be added to the QEMU commandline in the order they are encountered in the XML, *not* according to index (and what happens beyond that is anyway up to the decision of QEMU and the guest OS). So yes, definitely this topic needs to be clarified in the in-tree docs. I'm in the middle of something else right now, but will revisit this in a few days and try to come up with a patch that addresses this along with the previous 2 points.
(In reply to Laine Stump from comment #11) > (In reply to Michael S. Tsirkin from comment #9) > > Two points: > > 1. https://libvirt.org/pci-addresses.html > > makes it look like it enumerates all cases where bus numbers do not match. > > In my opinion, it looks like it's just providing some examples of where they > don't match in order to illustrate the fact that they don't always match, > and even says "that's [matching] not guaranteed to happen and will in fact > not be the case in all but the simplest scenarios". Yeah, that was the intention. > Andrea, I've Cc'ed you since you were the original author, and might have an > opinion... I believe we're on the same page, but then again we both have fairly intimate understanding of libvirt's PCI address allocation logic and so we're bound to take certain things for granted. This entire bug report is proof that our documentation needs to be expanded and improved upon. All the suggestions you've made so far look good.
Can reproduce this bug with libvirt-9.0.0-10.el9_2.x86_64 host version: kernel-5.14.0-307.el9.x86_64 qemu-kvm-8.0.0-2.el9.x86_64 seabios-1.16.1-1.el9.x86_64 guest: rhel.8.9.0 boot a guest with xml <domain type='kvm'> <name>gg</name> <uuid>4612b386-410c-11ec-b138-5405dbd35b7b</uuid> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>8388608</currentMemory> <vcpu placement='static'>3</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='4'/> <vcpupin vcpu='2' cpuset='6'/> <emulatorpin cpuset='12'/> </cputune> <numatune> <memory mode='strict' nodeset='0'/> </numatune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-rhel9.2.0'>hvm</type> <boot dev='hd'/> </os> <clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <pm> <suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source file='/home/RHEL-8.9-x86_64-latest.qcow2' index='1'/> <backingStore/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </disk> <controller type='usb' index='0' model='none'> <alias name='usb'/> </controller> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x10'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x11'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0x8'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0x9'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xa'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xb'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <controller type='sata' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0x8'/> <alias name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:bb:63:7b'/> <source bridge='switch'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </interface> <interface type='bridge'> <mac address='52:54:00:11:8f:ea'/> <source bridge='switch'/> <source dev='vnet1'/> <model type='virtio'/> <alias name='net1'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </interface> <interface type='bridge'> <mac address='52:54:00:11:8f:eb'/> <source bridge='switch'/> <source dev='vnet2'/> <model type='virtio'/> <alias name='net2'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/0'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <channel type='unix'> <source mode='bind' path='/tmp/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'> <alias name='input0'/> </input> <input type='keyboard' bus='ps2'> <alias name='input1'/> </input> <graphics type='vnc' port='5900' autoport='yes' listen='0.0.0.0'> <listen type='address' address='0.0.0.0'/> </graphics> <audio id='1' type='none'/> <video> <model type='cirrus' vram='16384' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </memballoon> </devices> <seclabel type='dynamic' model='selinux' relabel='yes'> <label>system_u:system_r:svirt_t:s0:c474,c598</label> <imagelabel>system_u:object_r:svirt_image_t:s0:c474,c598</imagelabel> </seclabel> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+107:+1001</label> <imagelabel>+107:+1001</imagelabel> </seclabel> </domain>
Note to QE: Please don't waste your time testing to see if this behavior has changed. As detailed in the earlier comments of this BZ, the behavior you are seeing is just how things work, and the only reason this BZ is still open is because the documentation could use some work to clarify that. Due to the relatively low priority of this (since most users don't pay any attention at all to the exact bus number of the pci devices in their guests :-)) I have ignored this BZ for a long time, but I guess I should spend some time to try and clarify the documentation so we can close the BZ.
Jiri & Laine - please let's work out what needs to be done documentation wise for this bug, so we can then just close it. I extended the stale/aging date to 30-Jun instead of 25-May just so that this doesn't close in the mean time.
After evaluating this issue, there are no plans to address it further or fix it in an upcoming release. Therefore, it is being closed. If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened.