Bug 1619884

Hi jingzhao,

During my test in libvirt, I found that only when there's no other pci device in the tested pcie-pci-bridge, we can't get pci device in guest after hotplug interface. Otherwise we can get pci device in guest. Please refer to https://bugzilla.redhat.com/show_bug.cgi?id=1520821#c21.

So can you help test the scenario that guest has another pci device(for example,-device intel-hda) in the same pcie-pci-bridge controller which we want to hotplug pci device to by qemu? 

Thank you very much.

Okay, after reading Bug 1520821 Comment 21, now I understand the real problem - it's a firmware (SeaBIOS/OVMF) or OS issue. My recollection is that when a PCI controller is initialized, if there are no devices plugged into it that require IO port space, then none will be allocated, and it isn't possible to add IO port space at a later time.

I don't know if there is an easy solution to this problem (although I would point out that since most legacy PCI devices require IO port space, and since it's unlikely anyone will manually add a pcie-to-pci-bridge if they don't intend to add a legacy PCI device, it is kind of a pointless exercise to have the pcie-to-pci-bridge initialized with no IO port space.

Assuming this is actually the cause of the problem, hopefully qemu can do something to force the firmware to give IO port space to the controller at initialization time even if no devices are attached, but I can't say for sure, as I have 0 direct knowledge of the associated code in QEMU, SeaBIOS, or OVMF.

It's only an issue if someone manually adds a pcie-to-pci-bridge to a guest with no legacy PCI devices. At the moment there is no management application that does this (I think OpenStack may add extra pcie-root-ports, but probably none of them have any explicit knowledge about the pcie-to-pci-bridge. So from that point of view I think it's fairly safe - the potential of encountering the error is there for a very small fraction of direct users of libvirt, but in practice it's not even possible for the vast majority of users to get into this situation (since the only time they will ever have a pcie-to-pci-bridge in their config is if management adds a legacy PCI device (while the guest is shutdown) which in turn causes libvirt to auto-add a pcie-to-pci-bridge; but in this case the bridge won't be empty at the time the guest is started, so hotplug of any new devices *will* work properly.

(I *think* that hotplug into an empty "pci-bridge" (the legacy-only version used on 440fx machinetypes when the root bus is full) *does* work, because slot 0 of a pci-bridge always has some sort of invisible device used for SHPC hotplug, and because of that the BIOS always reserves IO port space for the bridge.)

I wrote something in the doc text field; see if that's adequate.

My opinion is that documenting the limitation is sufficient for RHEL7. We may want to fix it in RHEL8 though (I'm never sure just how important hotplug is, and this is an even more limited case, since it is hotplugging a legacy PCI device into a more modern, PCIe-based guest (users could instead hotplug PCIe devices)

Plug pci devices into pcie-pci-bridge success，can not reproduce this bug.

host version:
qemu-kvm-2.12.0-65.module+el8.1.0+2983+b2ae9c0a.x86_64
kernel-4.18.0-80.el8.x86_64
seabios-1.11.1-3.module+el8+2529+a9686a4d.x86_64
guest:rhel8(kernel-4.18.0-80.el8.x86_64)

test steps:
1.Boot guest with cmd
/usr/libexec/qemu-kvm \
-M q35 \
-cpu SandyBridge \
-nodefaults -rtc base=utc \
-m 4G \
-smp 4,sockets=4,cores=1,threads=1 \
-enable-kvm \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-k en-us \
-nodefaults \
-boot menu=on \
-qmp tcp:0:6667,server,nowait \
-usb \
-device usb-tablet \
-vga qxl \
-device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x3 \
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x3.0x1 \
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x3.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x3.0x3,bus-reserve=1 \
-drive file=/home/rhel8.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,drive=drive-virtio-disk0,id=virtio-disk0,bus=pci.1,bootindex=0 \
-device virtio-net-pci,netdev=tap10,mac=00:52:68:26:31:03,id=net0,bus=pci.2 -netdev tap,id=tap10 \
-device pcie-pci-bridge,id=bridge1,bus=pci.3 \
-device pcie-root-port,port=0x16,chassis=7,id=pci.7,bus=pcie.0,addr=0x7 \
-device pcie-pci-bridge,id=bridge2,bus=pci.7 \
-device rtl8139,netdev=tap11,mac=00:52:68:26:31:04,id=net1,bus=bridge2,addr=0x1 -netdev tap,id=tap11 \
-device pcie-root-port,port=0x17,chassis=8,id=pci.8,bus=pcie.0,addr=0x9,bus-reserve=1 \
-device pcie-pci-bridge,id=bridge3,bus=pci.8 \
-monitor stdio \
-vnc :1 \


2. Hotplug pci device to pcie-pci-bridge

{"execute": "netdev_add", "arguments": { "type":"tap","id":"tap12"}}
{"return": {}}
{"execute": "device_add", "arguments": { "driver":"e1000","netdev":"tap12","mac":"22:11:22:45:61:98","id":"net2","bus":"bridge3","addr":"0x1"}}
{"return": {}}

3. Check device through "info qtree" or "info network"
(qemu) info network 
net0: index=0,type=nic,model=virtio-net-pci,macaddr=00:52:68:26:31:03
 \ tap10: index=0,type=tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
net1: index=0,type=nic,model=rtl8139,macaddr=00:52:68:26:31:04
 \ tap11: index=0,type=tap,ifname=tap1,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
net2: index=0,type=nic,model=e1000,macaddr=22:11:22:45:61:98
 \ tap12: index=0,type=tap,ifname=tap2,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown

4.Check pci device in the guest.

# lspci -v -t
-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
           +-01.0  Red Hat, Inc. QXL paravirtual graphic card
           +-03.0-[01]----00.0  Red Hat, Inc. Virtio block device
           +-03.1-[02]----00.0  Red Hat, Inc. Virtio network device
           +-03.2-[03-04]----00.0-[04]--
           +-03.3-[05-06]--
           +-07.0-[07-08]----00.0-[08]----01.0  Realtek Semiconductor Co., Ltd. RTL-8100/8101L/8139 PCI Fast Ethernet Adapter
           +-09.0-[09-0a]----00.0-[0a]----01.0  Intel Corporation 82540EM Gigabit Ethernet Controller
           +-1d.0  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1
           +-1d.1  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2
           +-1d.2  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3
           +-1d.7  Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1
           +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
           +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
           \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller

# ifconfig 
enp10s1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.73.227.22  netmask 255.255.252.0  broadcast 10.73.227.255
        inet6 fe80::2011:22ff:fe45:6198  prefixlen 64  scopeid 0x20<link>
        inet6 2620:52:0:49e0:2011:22ff:fe45:6198  prefixlen 64  scopeid 0x0<global>
        ether 22:11:22:45:61:98  txqueuelen 1000  (Ethernet)
        RX packets 2219  bytes 160853 (157.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 149  bytes 21667 (21.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp2s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.73.226.27  netmask 255.255.252.0  broadcast 10.73.227.255
        inet6 2620:52:0:49e0:252:68ff:fe26:3103  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::252:68ff:fe26:3103  prefixlen 64  scopeid 0x20<link>
        ether 00:52:68:26:31:03  txqueuelen 1000  (Ethernet)
        RX packets 2084  bytes 134870 (131.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 20  bytes 2388 (2.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp8s1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.73.224.238  netmask 255.255.252.0  broadcast 10.73.227.255
        inet6 2620:52:0:49e0:252:68ff:fe26:3104  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::252:68ff:fe26:3104  prefixlen 64  scopeid 0x20<link>
        ether 00:52:68:26:31:04  txqueuelen 1000  (Ethernet)
        RX packets 1871  bytes 94734 (92.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 10  bytes 1336 (1.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

add test results to Comment 8:

test results:
pci device can be found in the guest, and NIC can work well

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days