Bug 1390329

Martin/Amnon,

we need to decide whether we want to keep this device enabled for 7.6.

Mirek

Ass discussed over email, we can leave out PCIe-PCI bridge for now

Product Management has reviewed and declined this request.
You may appeal this decision by reopening this request.

Fix included in qemu-kvm-rhev-2.12.0-11.el7

Tested it with following test scenarios:

1. Check pci device that connected to pcie-pci-bridge directly

qemu command:

-device pcie-root-port,port=0x16,chassis=7,id=pci.7,bus=pcie.0,addr=0x7 \
-device pcie-pci-bridge,id=bridge2,bus=pci.7 \
-device rtl8139,netdev=tap11,mac=00:52:68:26:31:04,id=net1,bus=bridge2,addr=0x1 -netdev tap,id=tap11 \


Result: rtl8139 works well and ping other ip successfully

2. Check pci device that plug into pcie-pci-bridge

qemu command:

..........
-device pcie-root-port,port=0x13,chassis=4,id=pci.4,bus=pcie.0,addr=0x3.0x3,bus-reserve=1 \
-device pcie-pci-bridge,id=bridge1,bus=pci.3 \

hotplug command:

{"execute": "netdev_add", "arguments": { "type":"tap","id":"tap12"}}
{"return": {}}
{"execute": "device_add", "arguments": { "driver":"rtl8139","netdev":"tap12","mac":"22:11:22:45:61:98","id":"net2","bus":"bridge1","addr":"0x1"}}
{"return": {}}


Result:

hotplug failed, and device didn't find in guest although you can find through "info qtree" or "info network"

3. hotplug pcie-pci-bridge and then hotplug pci device

hotplug command:

(qemu) device_add pcie-pci-bridge,id=birdge3,bus=pci.8

{"execute": "netdev_add", "arguments": { "type":"tap","id":"tap12"}}
{"return": {}}
{"execute": "device_add", "arguments": { "driver":"e1000","netdev":"tap12","mac":"22:11:22:45:61:98","id":"net2","bus":"bridge1","addr":"0x1"}}
{"return": {}}

Result: the same with above scenario 2, detailed info, please see following comments

4.abnormal testing: hotplug pci device to pcie-pci-bridge that without "bus-reserver=1"

Result: guest reboot immediately after hotplug, and then you can find the pci device in guest

Question: "bus-reserver=1" must be add in "pcie-root-port" according to https://github.com/qemu/qemu/blob/master/docs/pcie_pci_bridge.txt, can we add a warning or other messages when we hotplug pcie-pci-bridge to pcie-root-port?


5.attach a virtio device to pcie-pci-bridge

Result: such as virtio-net-pci, works well. is it expected?

Hotplug failed detailed info:

1) version:

qemu-kvm-rhev-2.12.0-11.el7.x86_64
kernel-3.10.0-933.el7.x86_64

2) detailed steps:

1. Boot guest with qemu command [1]

2. Hotplug pcie-pci-bridge

(qemu) device_add pcie-pci-bridge,id=bridge3,bus=pci.8

3.Hotplug pci device to above pcie-pci-bridge

{"execute": "netdev_add", "arguments": { "type":"tap","id":"tap12"}}
{"return": {}}
{"execute": "device_add", "arguments": { "driver":"e1000","netdev":"tap12","mac":"22:11:22:45:61:98","id":"net2","bus":"bridge3","addr":"0x1"}}
{"return": {}}

4.Can found through "info network" or "info qtree"

(qemu) info network 
net0: index=0,type=nic,model=virtio-net-pci,macaddr=00:52:68:26:31:03
 \ tap10: index=0,type=tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
net1: index=0,type=nic,model=rtl8139,macaddr=00:52:68:26:31:04
 \ tap11: index=0,type=tap,ifname=tap1,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
net2: index=0,type=nic,model=e1000,macaddr=22:11:22:45:61:98
 \ tap12: index=0,type=tap,ifname=tap2,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
(qemu) info qtree
bus: main-system-bus
  type System
  dev: kvm-ioapic, id ""
    gpio-in "" 24
    gsi_base = 0 (0x0)
    mmio 00000000fec00000/0000000000001000
  dev: q35-pcihost, id ""
    MCFG = 2952790016 (0xb0000000)
    pci-hole64-size = 34359738368 (32 GiB)
    short_root_bus = 0 (0x0)
    below-4g-mem-size = 2147483648 (2 GiB)
    above-4g-mem-size = 2147483648 (2 GiB)
    x-pci-hole64-fix = true
    bus: pcie.0
      type PCIE
      dev: pcie-root-port, id "pci.8"
        x-migrate-msix = true
        bus-reserve = 1 (0x1)
        io-reserve = 18446744073709551615 (16 EiB)
        mem-reserve = 18446744073709551615 (16 EiB)
        pref32-reserve = 18446744073709551615 (16 EiB)
        pref64-reserve = 18446744073709551615 (16 EiB)
        power_controller_present = true
        chassis = 8 (0x8)
        slot = 0 (0x0)
        port = 23 (0x17)
        aer_log_max = 8 (0x8)
        addr = 09.0
        romfile = ""
        rombar = 1 (0x1)
        multifunction = false
        command_serr_enable = true
        x-pcie-lnksta-dllla = true
        x-pcie-extcap-init = true
        class PCI bridge, addr 00:09.0, pci id 1b36:000c (sub 0000:0000)
        bar 0: mem at 0xfce17000 [0xfce17fff]
        bus: pci.8
          type PCIE
          dev: pcie-pci-bridge, id "bridge3"
            msi = "auto"
            addr = 00.0
            romfile = ""
            rombar = 1 (0x1)
            multifunction = false
            command_serr_enable = true
            x-pcie-lnksta-dllla = true
            x-pcie-extcap-init = true
            class PCI bridge, addr 09:00.0, pci id 1b36:000e (sub 0000:0000)
            bar 0: mem at 0xfc500000 [0xfc5000ff]
            bus: bridge3
              type PCI
              dev: e1000, id "net2"
                mac = "22:11:22:45:61:98"
                vlan = <null>
                netdev = "tap12"
                autonegotiation = true
                mitigation = true
                extra_mac_registers = true
                migrate_tso_props = true
                addr = 01.0
                romfile = "pxe-e1000.rom"
                rombar = 1 (0x1)
                multifunction = false
                command_serr_enable = true
                x-pcie-lnksta-dllla = true
                x-pcie-extcap-init = true
                class Ethernet controller, addr 0a:01.0, pci id 8086:100e (sub 1af4:1100)
                bar 0: mem at 0xffffffffffffffff [0x1fffe]
                bar 1: i/o at 0xffffffffffffffff [0x3e]
                bar 6: mem at 0xffffffffffffffff [0x3fffe]

5. Check it in the guest, but didn't find the net device

[root@bootp-73-199-180 ~]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.73.199.180  netmask 255.255.252.0  broadcast 10.73.199.255
        inet6 2620:52:0:49c4:252:68ff:fe26:3104  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::252:68ff:fe26:3104  prefixlen 64  scopeid 0x20<link>
        ether 00:52:68:26:31:04  txqueuelen 1000  (Ethernet)
        RX packets 3184  bytes 245366 (239.6 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 426  bytes 51348 (50.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 00:52:68:26:31:03  txqueuelen 1000  (Ethernet)
        RX packets 3124  bytes 214404 (209.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 8  bytes 480 (480.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8  bytes 480 (480.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[root@bootp-73-199-180 ~]# lspci -vvv -t
-[0000:00]-+-00.0  Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller
           +-01.0  Red Hat, Inc. QXL paravirtual graphic card
           +-03.0-[01]----00.0  Red Hat, Inc. Virtio block device
           +-03.1-[02]----00.0  Red Hat, Inc. Virtio network device
           +-03.2-[03-04]----00.0-[04]--
           +-03.3-[05-06]--
           +-07.0-[07-08]----00.0-[08]----01.0  Realtek Semiconductor Co., Ltd. RTL-8100/8101L/8139 PCI Fast Ethernet Adapter
           +-09.0-[09-0a]----00.0-[0a]--
           +-1d.0  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #1
           +-1d.1  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #2
           +-1d.2  Intel Corporation 82801I (ICH9 Family) USB UHCI Controller #3
           +-1d.7  Intel Corporation 82801I (ICH9 Family) USB2 EHCI Controller #1
           +-1f.0  Intel Corporation 82801IB (ICH9) LPC Interface Controller
           +-1f.2  Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode]
           \-1f.3  Intel Corporation 82801I (ICH9 Family) SMBus Controller


6.dmesg of guest, please check the attachment

Created attachment 1477467 [details]
guest dmesg of hotplug

(In reply to jingzhao from comment #17)
> Tested it with following test scenarios:
> 
> 1. Check pci device that connected to pcie-pci-bridge directly
[...] 
> Result: rtl8139 works well and ping other ip successfully
> 
> 2. Check pci device that plug into pcie-pci-bridge
[...]
> Result:
> hotplug failed, and device didn't find in guest although you can find
> through "info qtree" or "info network"

This needs to be investigated.

> 
> 3. hotplug pcie-pci-bridge and then hotplug pci device
[...]
> Result: the same with above scenario 2, detailed info, please see following
> comments

Hotplugging the pcie-pci-bridge device itself is not supported, so this is expected.  I will investigate why it's not marked as not-hotpluggable upstream.


> 
> 4.abnormal testing: hotplug pci device to pcie-pci-bridge that without
> "bus-reserver=1"
> 
> Result: guest reboot immediately after hotplug, and then you can find the
> pci device in guest
> 
> Question: "bus-reserver=1" must be add in "pcie-root-port" according to
> https://github.com/qemu/qemu/blob/master/docs/pcie_pci_bridge.txt, can we
> add a warning or other messages when we hotplug pcie-pci-bridge to
> pcie-root-port?

I don't know the answer, this needs to be investigated.

> 
> 
> 5.attach a virtio device to pcie-pci-bridge
> 
> Result: such as virtio-net-pci, works well. is it expected?

Thanks for the results!

Note that hotplug of legacy PCI devices on Q35 will be tech preview, so these failures shouldn't prevent marking the BZ as verified, as long as the device works with the way libvirt expects it to work and doesn't cause regressions.  We can open a new BZ for the hotplug issues above (2 and 4).

Functional testing of PCI bridge with libvirt is as important as testing QEMU directly, to ensure there are no regressions and that libvirt is really going to use pcie-pci-bridge instead of the dmi bridge.  Is this part of the test plan?

(In reply to Eduardo Habkost from comment #22)
> (In reply to jingzhao from comment #17)
> > Tested it with following test scenarios:
> > 
> > 1. Check pci device that connected to pcie-pci-bridge directly
> [...] 
> > Result: rtl8139 works well and ping other ip successfully
> > 
> > 2. Check pci device that plug into pcie-pci-bridge
> [...]
> > Result:
> > hotplug failed, and device didn't find in guest although you can find
> > through "info qtree" or "info network"
> 
> This needs to be investigated.
> 
> > 
> > 3. hotplug pcie-pci-bridge and then hotplug pci device
> [...]
> > Result: the same with above scenario 2, detailed info, please see following
> > comments
> 
> Hotplugging the pcie-pci-bridge device itself is not supported, so this is
> expected.  I will investigate why it's not marked as not-hotpluggable
> upstream.

Hi Eduardo

According to line 81 of https://github.com/qemu/qemu/blob/master/docs/pcie_pci_bridge.txt, hotplug of pcie-pci-bridge is supported, If not support pcie-pci-bridge, should we file a new issue for tracking it?


> 
> 
> > 
> > 4.abnormal testing: hotplug pci device to pcie-pci-bridge that without
> > "bus-reserver=1"
> > 
> > Result: guest reboot immediately after hotplug, and then you can find the
> > pci device in guest
> > 
> > Question: "bus-reserver=1" must be add in "pcie-root-port" according to
> > https://github.com/qemu/qemu/blob/master/docs/pcie_pci_bridge.txt, can we
> > add a warning or other messages when we hotplug pcie-pci-bridge to
> > pcie-root-port?
> 
> I don't know the answer, this needs to be investigated.
> 
> > 
> > 
> > 5.attach a virtio device to pcie-pci-bridge
> > 
> > Result: such as virtio-net-pci, works well. is it expected?

PCIe devices that plug into pcie-pci-bridge work well, is it expected?


> 
> Thanks for the results!
> 
> Note that hotplug of legacy PCI devices on Q35 will be tech preview, so
> these failures shouldn't prevent marking the BZ as verified, as long as the
> device works with the way libvirt expects it to work and doesn't cause
> regressions.  We can open a new BZ for the hotplug issues above (2 and 4).

Please check bz1619884  & bz 1619885

> 
> Functional testing of PCI bridge with libvirt is as important as testing
> QEMU directly, to ensure there are no regressions and that libvirt is really
> going to use pcie-pci-bridge instead of the dmi bridge.  Is this part of the
> test plan?
I just focus on testing of qemu part. and bz1590632 is focus on libvirt part

Another question: only 1 pci device can plug into pcie-pci-bridge or support 2 or more devices can be plug into pcie-pci-bridge, and I tried it and 2 rtl8139 devices can be plug into the same pcie-pci-bridge. also, if support much 2 devices pluginto the same pcie-pci-bridge, the upper limmit (such as 32 or much more)? could you help to confirm it?


Thanks
Jing

(In reply to jingzhao from comment #23)

> 
> According to line 81 of
> https://github.com/qemu/qemu/blob/master/docs/pcie_pci_bridge.txt, hotplug
> of pcie-pci-bridge is supported, If not support pcie-pci-bridge, should we
> file a new issue for tracking it?


If you're just intending to test everything that's in the qemu binary, and you're doing it by hand-creating your qemu commandlines, then you'll want to add bus-reserve=1 to the commandline for the pcie-root-port device that will have the pcie-pci-bridge plugged into it.

If, on the other hand, your interest is only in testing everything that is possible via the Red Hat supported method of controlling qemu (which is "only via libvirt") then you don't need to test hotplug of the controller itself, because libvirt doesn't support the hotplug of *any* pci controller, including pcie-pci-bridge.

If you want libvirt to add support for hotplugging this controller, then you should file an RFE (this is not a bug) and have it evaluated to decide if it's worthwhile to add. Note that until today, libvirt has never supported the hotplug of *any* PCI controller. Bug 1520821 Comment 4 contains a very good summary of why libvirt thinks it is not worthwhile to support hotplug of this device.


> 
> 
> > 
> > 
> > > 
> > > 4.abnormal testing: hotplug pci device to pcie-pci-bridge that without
> > > "bus-reserver=1"
> > > 
> > > Result: guest reboot immediately after hotplug, and then you can find the
> > > pci device in guest
> > > 
> > > Question: "bus-reserver=1" must be add in "pcie-root-port" according to
> > > https://github.com/qemu/qemu/blob/master/docs/pcie_pci_bridge.txt, can we
> > > add a warning or other messages when we hotplug pcie-pci-bridge to
> > > pcie-root-port?
> > 
> > I don't know the answer, this needs to be investigated.

Since libvirt doesn't support such a hotplug, it seems like a lot of work that will end up being meaningless in RHEL-based deployments of qemu (since it will never occur)

> > 
> > > 
> > > 
> > > 5.attach a virtio device to pcie-pci-bridge
> > > 
> > > Result: such as virtio-net-pci, works well. is it expected?
> 
> PCIe devices that plug into pcie-pci-bridge work well, is it expected?

Sure - in general it will just be missing the extended capabilities, similar to what happens if you connect a PCIe device to pcie-root on the initial commandline.

For virtio devices, it actually becomes a legacy PCI device, and this is *very important*, because when a virtio device is plugged into a legacy PCI slot, it has the legacy virtio mode (virtio-0.9) enabled, and that is the only way that a virtio device will be recognized by the virtio device drivers in a RHEL6 guest. (a virtio device plugged into a standard PCIe port will only work with a guest driver that is aware of virtio-1.0 - RHEL6's driver isn't.)


> Another question: only 1 pci device can plug into pcie-pci-bridge or support
> 2 or more devices can be plug into pcie-pci-bridge, and I tried it and 2
> rtl8139 devices can be plug into the same pcie-pci-bridge. also, if support
> much 2 devices pluginto the same pcie-pci-bridge, the upper limmit (such as
> 32 or much more)? could you help to confirm it?

The pcie-pci-bridge is similar to the pcie-root, pci-bridge, and dmi-to-pci-bridge controllers in that is allows slots numbered 0-31 (slot 0 is often reserved, but in this case it is not). Each slot has 8 functions. This all means that you could conceivably connect 8x32 = 256 devices to a single pcie-pci-bridge if you did it on the commandline, or you could hotplug 32 devices into an empty pcie-pci-bridge after qemu is running (hotplugging multiple functions is tricky if it even works at all.)

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:3443