Bug 2166247

Summary: the pci addresses of qmp and guest doesn't equal
Product: Red Hat Enterprise Linux 9 Reporter: Han Han <hhan>
Component: qemu-kvmAssignee: Michael S. Tsirkin <mst>
qemu-kvm sub component: PCI QA Contact: Yiqian Wei <yiwei>
Status: CLOSED NOTABUG Docs Contact:
Severity: high    
Priority: high CC: coli, jinzhao, juzhang, leiyang, qcheng, qinwang, virt-maint, xuwei, zhguo
Version: 9.2   
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-02-10 08:43:36 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
VM XML, qemu cmdline none

Description Han Han 2023-02-01 09:45:38 UTC 
Created attachment 1941488 [details]
VM XML, qemu cmdline

Description of problem:
As subject

Version-Release number of selected component (if applicable):
host:
libvirt-9.0.0-2.el9.x86_64
qemu-kvm-7.2.0-5.el9.x86_64

guest: kernel-5.14.0-251.el9.x86_64

How reproducible:
100%

Steps to Reproduce:
1. Start a VM with the vsock XML
<vsock model="virtio">
  <cid auto="yes"/>
  <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
</vsock>

QEMU cmdline for vsock:
/usr/libexec/qemu-kvm ... -device {"driver":"vhost-vsock-pci","id":"vsock0","guest-cid":3,"vhostfd":"20","bus":"pci.7","addr":"0x0"} ...

2. Check vsock in guest by lspci
[root@localhost ~]# lspci|grep sock
08:00.0 Communication controller: Red Hat, Inc. Virtio socket (rev 01)


The PCI addresses are not equal.

Actual results:
As above

Expected results:
The PCI addresses are equal.

Additional info:
For VM XML, qemu cmdline, please check them in the attachment.
It could also be reproduced by hot-plug.
Comment 1 qing.wang 2023-02-09 07:39:07 UTC 
xml:
<domain type='kvm'>
  <name>rhel-9.2-new</name>
  <uuid>fa79b2f5-2fa3-40da-932f-96631fba4761</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://libosinfo.org/unknown"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static' current='1'>8</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-rhel9.2.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'/>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/rhel-9.2.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x0A' slot='0x00' function='0x0'/>
    </disk>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x16'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0x17'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x7'/>
    </controller>
    <controller type='pci' index='9' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='9' port='0x18'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='10' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='10' port='0x19'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x1'/>
    </controller>
    <controller type='pci' index='11' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='11' port='0x1a'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/>
    </controller>
    <controller type='pci' index='12' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='12' port='0x1b'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/>
    </controller>
    <controller type='pci' index='13' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='13' port='0x1c'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/>
    </controller>
    <controller type='pci' index='14' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='14' port='0x1d'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/>
    </controller>
    <controller type='pci' index='15' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='15' port='0x1e'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x6'/>
    </controller>
    <controller type='pci' index='16' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='qemu-xhci'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:aa:2b:86'/>
      <source network='default'/>
      <model type='e1000e'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='dev'>
      <source path='/dev/ttyS0'/>
      <target type='isa-serial' port='2'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <graphics type='vnc' port='5905' listen='0.0.0.0'/>
    <console type='dev'>
      <source path='/dev/ttyS0'/>
      <target type='serial' port='2'/>
    </console>
    <channel type='pty'>
      <target type='virtio' name='org.linux-kvm.port.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='3'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <audio id='1' type='none'/>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </memballoon>
    <vsock model='virtio'>
      <cid auto='yes'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </vsock>
  </devices>
</domain>



qemu cmd line
/usr/libexec/qemu-kvm \
-name guest=rhel-9.2-new,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-4-rhel-9.2-new/master-key.aes"}' \
-machine pc-q35-rhel9.2.0,usb=off,dump-guest-core=off,memory-backend=pc.ram \
-accel kvm \
-cpu host,migratable=on \
-m 2048 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":2147483648}' \
-overcommit mem-lock=off \
-smp 1,maxcpus=8,sockets=8,cores=1,threads=1 \
-uuid fa79b2f5-2fa3-40da-932f-96631fba4761 \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=33,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-global ICH9-LPC.disable_s3=1 \
-global ICH9-LPC.disable_s4=1 \
-boot strict=on \
-device '{"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \
-device '{"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"}' \
-device '{"driver":"pcie-root-port","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"}' \
-device '{"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"}' \
-device '{"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x2.0x4"}' \
-device '{"driver":"pcie-root-port","port":21,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x2.0x5"}' \
-device '{"driver":"pcie-root-port","port":22,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x2.0x6"}' \
-device '{"driver":"pcie-root-port","port":23,"chassis":8,"id":"pci.8","bus":"pcie.0","addr":"0x2.0x7"}' \
-device '{"driver":"pcie-root-port","port":24,"chassis":9,"id":"pci.9","bus":"pcie.0","multifunction":true,"addr":"0x3"}' \
-device '{"driver":"pcie-root-port","port":25,"chassis":10,"id":"pci.10","bus":"pcie.0","addr":"0x3.0x1"}' \
-device '{"driver":"pcie-root-port","port":26,"chassis":11,"id":"pci.11","bus":"pcie.0","addr":"0x3.0x2"}' \
-device '{"driver":"pcie-root-port","port":27,"chassis":12,"id":"pci.12","bus":"pcie.0","addr":"0x3.0x3"}' \
-device '{"driver":"pcie-root-port","port":28,"chassis":13,"id":"pci.13","bus":"pcie.0","addr":"0x3.0x4"}' \
-device '{"driver":"pcie-root-port","port":29,"chassis":14,"id":"pci.14","bus":"pcie.0","addr":"0x3.0x5"}' \
-device '{"driver":"pcie-root-port","port":30,"chassis":15,"id":"pci.15","bus":"pcie.0","addr":"0x3.0x6"}' \
-device '{"driver":"pcie-pci-bridge","id":"pci.16","bus":"pci.4","addr":"0x0"}' \
-device '{"driver":"qemu-xhci","id":"usb","bus":"pci.3","addr":"0x0"}' \
-device '{"driver":"virtio-scsi-pci","id":"scsi0","bus":"pci.2","addr":"0x0"}' \
-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.5","addr":"0x0"}' \
-blockdev '{"driver":"file","filename":"/home/rhel-9.2.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"driver":"qcow2","file":"libvirt-2-storage","backing":null}' \
-device '{"driver":"virtio-blk-pci","bus":"pci.10","addr":"0x0","drive":"libvirt-2-format","id":"virtio-disk0","bootindex":1}' \
-blockdev '{"driver":"file","filename":"/home/kvm_autotest_root/images/mstg2.qcow2","node-name":"libvirt-1-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"discard":"unmap","cache":{"direct":true,"no-flush":false},"driver":"qcow2","file":"libvirt-1-storage","backing":null}' \
-device '{"driver":"scsi-hd","bus":"scsi0.0","channel":0,"scsi-id":0,"lun":1,"device_id":"drive-scsi0-0-0-1","drive":"libvirt-1-format","id":"scsi0-0-0-1","write-cache":"on"}' \
-netdev '{"type":"tap","fd":"34","id":"hostnet0"}' \
-device '{"driver":"e1000e","netdev":"hostnet0","id":"net0","mac":"52:54:00:aa:2b:86","bus":"pci.1","addr":"0x0"}' \
-chardev serial,id=charserial0,path=/dev/ttyS0 \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":2}' \
-chardev pty,id=charchannel0 \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":3,"chardev":"charchannel0","id":"channel0","name":"org.linux-kvm.port.0"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-vnc 0.0.0.0:5,audiodev=audio1 \
-device '{"driver":"cirrus-vga","id":"video0","bus":"pcie.0","addr":"0x1"}' \
-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.6","addr":"0x0"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-device '{"driver":"vhost-vsock-pci","id":"vsock0","guest-cid":3,"vhostfd":"29","bus":"pci.7","addr":"0x0"}' \
-msg timestamp=on


lspci in guest:
00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
00:01.0 VGA compatible controller: Cirrus Logic GD 5446
00:02.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:02.1 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:02.2 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:02.3 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:02.4 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:02.5 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:02.6 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:02.7 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:03.0 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:03.1 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:03.2 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:03.3 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:03.4 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:03.5 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:03.6 PCI bridge: Red Hat, Inc. QEMU PCIe Root port
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
02:00.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI (rev 01)
03:00.0 USB controller: Red Hat, Inc. QEMU XHCI Host Controller (rev 01)
04:00.0 PCI bridge: Red Hat, Inc. Device 000e
06:00.0 Communication controller: Red Hat, Inc. Virtio console (rev 01)
07:00.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon (rev 01)
08:00.0 Communication controller: Red Hat, Inc. Virtio socket (rev 01)
0b:00.0 SCSI storage controller: Red Hat, Inc. Virtio block device (rev 01)

It looks like have 1 offset for the pci assignment
Comment 2 qing.wang 2023-02-10 02:14:38 UTC 
[root@localhost ~]# lspci -tv
-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
           +-01.0  Cirrus Logic GD 5446
           +-02.0-[01]----00.0  Intel Corporation 82574L Gigabit Network Connection
           +-02.1-[02]----00.0  Red Hat, Inc. Virtio SCSI
           +-02.2-[03]----00.0  Red Hat, Inc. QEMU XHCI Host Controller
           +-02.3-[04-05]----00.0-[05]--
           +-02.4-[06]----00.0  Red Hat, Inc. Virtio console
           +-02.5-[07]----00.0  Red Hat, Inc. Virtio memory balloon
           +-02.6-[08]----00.0  Red Hat, Inc. Virtio socket
           +-02.7-[09]--
           +-03.0-[0a]--
           +-03.1-[0b]----00.0  Red Hat, Inc. Virtio block device
           +-03.2-[0c]--
           +-03.3-[0d]--
           +-03.4-[0e]--
           +-03.5-[0f]--
           +-03.6-[10]--
           +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
           +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
           \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller
[root@localhost ~]#
Comment 3 Michael S. Tsirkin 2023-02-10 08:43:36 UTC 
see:
https://libvirt.org/formatdomain.html

The relationship between the PCI addresses configured in the domain XML and those seen by the guest OS can sometime seem confusing: a separate document describes how PCI addresses work in more detail.

and

https://libvirt.org/pci-addresses.html