Bug 1619884 - [Q35]Plug pci devices into pcie-pci-bridge failed [NEEDINFO]
Summary: [Q35]Plug pci devices into pcie-pci-bridge failed
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: qemu-kvm
Version: 8.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Michael S. Tsirkin
QA Contact: jingzhao
Jiri Herrmann
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-08-22 02:06 UTC by jingzhao
Modified: 2019-07-26 11:45 UTC (History)
15 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
.Hot-plugging PCI devices to a pcie-to-pci bridge controller works correctly Previously, if a guest virtual machine configuration contained a pcie-to-pci-bridge controller that had no endpoint devices attached to it at the time the guest was started, hot-plugging new devices to that controller was not possible. This update improves how hot-plugging legacy PCI devices on a PCIe system is handled, which prevents the problem from occurring.
Clone Of:
Environment:
Last Closed: 2019-04-11 07:39:15 UTC
Type: Bug
Target Upstream Version:
jherrman: needinfo? (mst)


Attachments (Terms of Use)
dmesg log of guest (49.96 KB, text/plain)
2018-08-22 02:06 UTC, jingzhao
no flags Details

Description jingzhao 2018-08-22 02:06:34 UTC
Created attachment 1477780 [details]
dmesg log of guest

Description of problem:
Hotplug failed and device didn't find in the guest although you can find through "info qtree" or "info network"

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.12.0-11.el7.x86_64
kernel-3.10.0-933.el7.x86_64

How reproducible:
3/3

Steps to Reproduce:
1.Boot guest with qemu command line [1]

2. Hotplug pci device to pcie-pci-bridge

{"execute": "netdev_add", "arguments": { "type":"tap","id":"tap12"}}
{"return": {}}
{"execute": "device_add", "arguments": { "driver":"e1000","netdev":"tap12","mac":"22:11:22:45:61:98","id":"net2","bus":"bridge3","addr":"0x1"}}
{"return": {}}

3. Check device through "info qtree" or "info network"

(qemu) info qtree
bus: main-system-bus
  type System
  dev: kvm-ioapic, id ""
    gpio-in "" 24
    gsi_base = 0 (0x0)
    mmio 00000000fec00000/0000000000001000
  dev: q35-pcihost, id ""
    MCFG = 2952790016 (0xb0000000)
    pci-hole64-size = 34359738368 (32 GiB)
    short_root_bus = 0 (0x0)
    below-4g-mem-size = 2147483648 (2 GiB)
    above-4g-mem-size = 2147483648 (2 GiB)
    x-pci-hole64-fix = true
    bus: pcie.0
      type PCIE
      dev: pcie-root-port, id "pci.8"
        x-migrate-msix = true
        bus-reserve = 1 (0x1)
        io-reserve = 18446744073709551615 (16 EiB)
        mem-reserve = 18446744073709551615 (16 EiB)
        pref32-reserve = 18446744073709551615 (16 EiB)
        pref64-reserve = 18446744073709551615 (16 EiB)
        power_controller_present = true
        chassis = 8 (0x8)
        slot = 0 (0x0)
        port = 23 (0x17)
        aer_log_max = 8 (0x8)
        addr = 09.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        x-pcie-lnksta-dllla = true
        x-pcie-extcap-init = true
        class PCI bridge, addr 00:09.0, pci id 1b36:000c (sub 0000:0000)
        bar 0: mem at 0xfce17000 [0xfce17fff]
        bus: pci.8
          type PCIE
          dev: pcie-pci-bridge, id "bridge3"
            msi = "auto"
            addr = 00.0
            romfile = ""
            rombar = 1 (0x1)
            multifunction = false
            command_serr_enable = true
            x-pcie-lnksta-dllla = true
            x-pcie-extcap-init = true
            class PCI bridge, addr 09:00.0, pci id 1b36:000e (sub 0000:0000)
            bar 0: mem at 0xfc400000 [0xfc4000ff]
            bus: bridge3
              type PCI
              dev: e1000, id "net2"
                mac = "22:11:22:45:61:98"
                vlan = <null>
                netdev = "tap12"
                autonegotiation = true
                mitigation = true
                extra_mac_registers = true
                migrate_tso_props = true
                addr = 01.0
                romfile = "pxe-e1000.rom"
                rombar = 1 (0x1)
                multifunction = false
                command_serr_enable = true
                x-pcie-lnksta-dllla = true
                x-pcie-extcap-init = true
                class Ethernet controller, addr 0a:01.0, pci id 8086:100e (sub 1af4:1100)
                bar 0: mem at 0xffffffffffffffff [0x1fffe]
                bar 1: i/o at 0xffffffffffffffff [0x3e]
                bar 6: mem at 0xffffffffffffffff [0x3fffe]

(qemu) info network 
net0: index=0,type=nic,model=virtio-net-pci,macaddr=00:52:68:26:31:03
 \ tap10: index=0,type=tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
net1: index=0,type=nic,model=rtl8139,macaddr=00:52:68:26:31:04
 \ tap11: index=0,type=tap,ifname=tap1,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
net2: index=0,type=nic,model=e1000,macaddr=22:11:22:45:61:98
 \ tap12: index=0,type=tap,ifname=tap2,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown

4. Check pci device in the guest

# lspci -vvv -t
-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
           +-01.0  Red Hat, Inc. QXL paravirtual graphic card
           +-03.0-[01]----00.0  Red Hat, Inc. Virtio block device
           +-03.1-[02]----00.0  Red Hat, Inc. Virtio network device
           +-03.2-[03-04]----00.0-[04]--
           +-03.3-[05-06]--
           +-07.0-[07-08]----00.0-[08]----01.0  Realtek Semiconductor Co., Ltd. RTL-8100/8101L/8139 PCI Fast Ethernet Adapter
           +-09.0-[09-0a]----00.0-[0a]--
           +-1d.0  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1
           +-1d.1  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2
           +-1d.2  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3
           +-1d.7  Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1
           +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
           +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
           \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller

# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.73.199.180  netmask 255.255.252.0  broadcast 10.73.199.255
        inet6 2620:52:0:49c4:252:68ff:fe26:3104  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::252:68ff:fe26:3104  prefixlen 64  scopeid 0x20<link>
        ether 00:52:68:26:31:04  txqueuelen 1000  (Ethernet)
        RX packets 798  bytes 57123 (55.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 196  bytes 27349 (26.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 00:52:68:26:31:03  txqueuelen 1000  (Ethernet)
        RX packets 705  bytes 44250 (43.2 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 8  bytes 480 (480.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8  bytes 480 (480.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


Actual results:
Guest didn't find the pci device, demsg of guest, please check the attachment

Expected results:
pci device can be found in the guest, and work well

Additional info:
[1]
/usr/libexec/qemu-kvm \
-M q35 \
-cpu SandyBridge \
-nodefaults -rtc base=utc \
-m 4G \
-smp 4,sockets=4,cores=1,threads=1 \
-enable-kvm \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-k en-us \
-nodefaults \
-boot menu=on \
-qmp tcp:0:6667,server,nowait \
-usb \
-device usb-tablet \
-vga qxl \
-device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x3 \
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x3.0x1 \
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x3.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x3.0x3,bus-reserve=1 \
-drive file=/home/env/rhel76-64-virtio-scsi.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,drive=drive-virtio-disk0,id=virtio-disk0,bus=pci.1,bootindex=0 \
-device virtio-net-pci,netdev=tap10,mac=00:52:68:26:31:03,id=net0,bus=pci.2 -netdev tap,id=tap10 \
-device pcie-pci-bridge,id=bridge1,bus=pci.3 \
-device pcie-root-port,port=0x16,chassis=7,id=pci.7,bus=pcie.0,addr=0x7 \
-device pcie-pci-bridge,id=bridge2,bus=pci.7 \
-device rtl8139,netdev=tap11,mac=00:52:68:26:31:04,id=net1,bus=bridge2,addr=0x1 -netdev tap,id=tap11 \
-device pcie-root-port,port=0x17,chassis=8,id=pci.8,bus=pcie.0,addr=0x9,bus-reserve=1 \
-device pcie-pci-bridge,id=bridge3,bus=pci.8 \
-monitor stdio \
-vnc :1 \

Comment 4 Meina Li 2018-08-29 08:44:07 UTC
Hi jingzhao,

During my test in libvirt, I found that only when there's no other pci device in the tested pcie-pci-bridge, we can't get pci device in guest after hotplug interface. Otherwise we can get pci device in guest. Please refer to https://bugzilla.redhat.com/show_bug.cgi?id=1520821#c21.

So can you help test the scenario that guest has another pci device(for example,-device intel-hda) in the same pcie-pci-bridge controller which we want to hotplug pci device to by qemu? 

Thank you very much.

Comment 5 Laine Stump 2018-08-30 19:11:34 UTC
Okay, after reading Bug 1520821 Comment 21, now I understand the real problem - it's a firmware (SeaBIOS/OVMF) or OS issue. My recollection is that when a PCI controller is initialized, if there are no devices plugged into it that require IO port space, then none will be allocated, and it isn't possible to add IO port space at a later time.

I don't know if there is an easy solution to this problem (although I would point out that since most legacy PCI devices require IO port space, and since it's unlikely anyone will manually add a pcie-to-pci-bridge if they don't intend to add a legacy PCI device, it is kind of a pointless exercise to have the pcie-to-pci-bridge initialized with no IO port space.

Assuming this is actually the cause of the problem, hopefully qemu can do something to force the firmware to give IO port space to the controller at initialization time even if no devices are attached, but I can't say for sure, as I have 0 direct knowledge of the associated code in QEMU, SeaBIOS, or OVMF.

Comment 7 Laine Stump 2018-09-05 16:08:46 UTC
It's only an issue if someone manually adds a pcie-to-pci-bridge to a guest with no legacy PCI devices. At the moment there is no management application that does this (I think OpenStack may add extra pcie-root-ports, but probably none of them have any explicit knowledge about the pcie-to-pci-bridge. So from that point of view I think it's fairly safe - the potential of encountering the error is there for a very small fraction of direct users of libvirt, but in practice it's not even possible for the vast majority of users to get into this situation (since the only time they will ever have a pcie-to-pci-bridge in their config is if management adds a legacy PCI device (while the guest is shutdown) which in turn causes libvirt to auto-add a pcie-to-pci-bridge; but in this case the bridge won't be empty at the time the guest is started, so hotplug of any new devices *will* work properly.

(I *think* that hotplug into an empty "pci-bridge" (the legacy-only version used on 440fx machinetypes when the root bus is full) *does* work, because slot 0 of a pci-bridge always has some sort of invisible device used for SHPC hotplug, and because of that the BIOS always reserves IO port space for the bridge.)

I wrote something in the doc text field; see if that's adequate.

My opinion is that documenting the limitation is sufficient for RHEL7. We may want to fix it in RHEL8 though (I'm never sure just how important hotplug is, and this is an even more limited case, since it is hotplugging a legacy PCI device into a more modern, PCIe-based guest (users could instead hotplug PCIe devices)

Comment 8 Yiqian Wei 2019-04-11 06:50:09 UTC
Plug pci devices into pcie-pci-bridge success,can not reproduce this bug.

host version:
qemu-kvm-2.12.0-65.module+el8.1.0+2983+b2ae9c0a.x86_64
kernel-4.18.0-80.el8.x86_64
seabios-1.11.1-3.module+el8+2529+a9686a4d.x86_64
guest:rhel8(kernel-4.18.0-80.el8.x86_64)

test steps:
1.Boot guest with cmd
/usr/libexec/qemu-kvm \
-M q35 \
-cpu SandyBridge \
-nodefaults -rtc base=utc \
-m 4G \
-smp 4,sockets=4,cores=1,threads=1 \
-enable-kvm \
-uuid 990ea161-6b67-47b2-b803-19fb01d30d12 \
-k en-us \
-nodefaults \
-boot menu=on \
-qmp tcp:0:6667,server,nowait \
-usb \
-device usb-tablet \
-vga qxl \
-device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x3 \
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x3.0x1 \
-device pcie-root-port,port=0x12,chassis=3,id=pci.3,bus=pcie.0,addr=0x3.0x2 \
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x3.0x3,bus-reserve=1 \
-drive file=/home/rhel8.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none,werror=stop,rerror=stop \
-device virtio-blk-pci,drive=drive-virtio-disk0,id=virtio-disk0,bus=pci.1,bootindex=0 \
-device virtio-net-pci,netdev=tap10,mac=00:52:68:26:31:03,id=net0,bus=pci.2 -netdev tap,id=tap10 \
-device pcie-pci-bridge,id=bridge1,bus=pci.3 \
-device pcie-root-port,port=0x16,chassis=7,id=pci.7,bus=pcie.0,addr=0x7 \
-device pcie-pci-bridge,id=bridge2,bus=pci.7 \
-device rtl8139,netdev=tap11,mac=00:52:68:26:31:04,id=net1,bus=bridge2,addr=0x1 -netdev tap,id=tap11 \
-device pcie-root-port,port=0x17,chassis=8,id=pci.8,bus=pcie.0,addr=0x9,bus-reserve=1 \
-device pcie-pci-bridge,id=bridge3,bus=pci.8 \
-monitor stdio \
-vnc :1 \


2. Hotplug pci device to pcie-pci-bridge

{"execute": "netdev_add", "arguments": { "type":"tap","id":"tap12"}}
{"return": {}}
{"execute": "device_add", "arguments": { "driver":"e1000","netdev":"tap12","mac":"22:11:22:45:61:98","id":"net2","bus":"bridge3","addr":"0x1"}}
{"return": {}}

3. Check device through "info qtree" or "info network"
(qemu) info network 
net0: index=0,type=nic,model=virtio-net-pci,macaddr=00:52:68:26:31:03
 \ tap10: index=0,type=tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
net1: index=0,type=nic,model=rtl8139,macaddr=00:52:68:26:31:04
 \ tap11: index=0,type=tap,ifname=tap1,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
net2: index=0,type=nic,model=e1000,macaddr=22:11:22:45:61:98
 \ tap12: index=0,type=tap,ifname=tap2,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown

4.Check pci device in the guest.

# lspci -v -t
-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
           +-01.0  Red Hat, Inc. QXL paravirtual graphic card
           +-03.0-[01]----00.0  Red Hat, Inc. Virtio block device
           +-03.1-[02]----00.0  Red Hat, Inc. Virtio network device
           +-03.2-[03-04]----00.0-[04]--
           +-03.3-[05-06]--
           +-07.0-[07-08]----00.0-[08]----01.0  Realtek Semiconductor Co., Ltd. RTL-8100/8101L/8139 PCI Fast Ethernet Adapter
           +-09.0-[09-0a]----00.0-[0a]----01.0  Intel Corporation 82540EM Gigabit Ethernet Controller
           +-1d.0  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1
           +-1d.1  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2
           +-1d.2  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3
           +-1d.7  Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1
           +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
           +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
           \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller

# ifconfig 
enp10s1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.73.227.22  netmask 255.255.252.0  broadcast 10.73.227.255
        inet6 fe80::2011:22ff:fe45:6198  prefixlen 64  scopeid 0x20<link>
        inet6 2620:52:0:49e0:2011:22ff:fe45:6198  prefixlen 64  scopeid 0x0<global>
        ether 22:11:22:45:61:98  txqueuelen 1000  (Ethernet)
        RX packets 2219  bytes 160853 (157.0 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 149  bytes 21667 (21.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp2s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.73.226.27  netmask 255.255.252.0  broadcast 10.73.227.255
        inet6 2620:52:0:49e0:252:68ff:fe26:3103  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::252:68ff:fe26:3103  prefixlen 64  scopeid 0x20<link>
        ether 00:52:68:26:31:03  txqueuelen 1000  (Ethernet)
        RX packets 2084  bytes 134870 (131.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 20  bytes 2388 (2.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp8s1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.73.224.238  netmask 255.255.252.0  broadcast 10.73.227.255
        inet6 2620:52:0:49e0:252:68ff:fe26:3104  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::252:68ff:fe26:3104  prefixlen 64  scopeid 0x20<link>
        ether 00:52:68:26:31:04  txqueuelen 1000  (Ethernet)
        RX packets 1871  bytes 94734 (92.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 10  bytes 1336 (1.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Comment 9 Yiqian Wei 2019-04-11 07:39:15 UTC
add test results to Comment 8:

test results:
pci device can be found in the guest, and NIC can work well


Note You need to log in before you can comment on or make changes to this bug.