Created attachment 1824994 [details]
instance pc-q35-rhel8.3.0 xml

Created attachment 1824995 [details]
qemu instance logs

I'm moving this to qemu-kvm component and marking a regression, since the config provided works fine in QEMU 6.0.0 RPMs and fails with 6.1.0

Created attachment 1825000 [details]
instance pc-i440fx machine

For what it's worth, it looks like i440fx machine type doesn't have any pcie-root-port as opposed to pc-q35 ones [1].

[1]
~~~
$ grep -c "model='pcie-root-port'" instance*.xml 
instance-pci440fx.xml:0
instance.xml:17
~~~

This is fixed by setting

  <qemu:commandline>
    <qemu:arg value='-global'/>
    <qemu:arg value='ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off'/>
  </qemu:commandline>

aka

  -global ICH9-LPC.acpi-pci-hotplug-with-bridge-support=off


IOW, looks like the change in PCI hotplug impl has broken the ability to run guests with many pcie-root-ports.

> - When setting /etc/nova/nova.conf libvirt num_pcie_ports 4, there's no issue. (default value is 16)

The threshold for seeing trouble appears to be '15' - ie it works with 14 pcie-root-ports, fails with 15 pcie-root-ports .

I had been told in a discussion a couple weeks ago that the QEMU change in defaults to enable ACPI hotplug on Q35 only affected new q35 machinetypes, not existing q35 machinetypes (I asked because, although I'm completely unfamiliar with QEMU source, I didn't see anything in the patch switching the default to indicate it was only on new machinetypes; I wrote it off to my ignorance of the source).

Since the config specifically says "pc-q35-rhel8.3.0", I guess that's not the case, and the change in default takes effect for *all* Q35 machinetypes? Isn't that a violation of guest ABI, which could potentially break cross-version migration?

(In reply to Laine Stump from comment #7)
> I had been told in a discussion a couple weeks ago that the QEMU change in
> defaults to enable ACPI hotplug on Q35 only affected new q35 machinetypes,
> not existing q35 machinetypes (I asked because, although I'm completely
> unfamiliar with QEMU source, I didn't see anything in the patch switching
> the default to indicate it was only on new machinetypes; I wrote it off to
> my ignorance of the source).

What you were told is correct for *upstream* machine types. The
pc-q35-6.1.0 machine type references pc_compat_6_0 which contains
the setting { "ICH9-LPC", "acpi-pci-hotplug-with-bridge-support", "off" },

It would be correct for RHEL machine types too, except that we haven't
actually done the RHEL machine types when rebasing and I don't seee
anything referencing pc_compat_6_0 which is needed to ensure old hotplug
for old machine types.

> Since the config specifically says "pc-q35-rhel8.3.0", I guess that's not
> the case, and the change in default takes effect for *all* Q35 machinetypes?
> Isn't that a violation of guest ABI, which could potentially break
> cross-version migration?

The test was originally using pc-q35-rhel8.5.0 and we suspected this hotplug 
change, so switched to pc-q35-rhel8.3.0. This is when we discovered that the
 6.1.0 rebase had broken all existing machine types because it didn't wire 
up back compat yet.

(In reply to Daniel Berrangé from comment #6)
> > - When setting /etc/nova/nova.conf libvirt num_pcie_ports 4, there's no issue. (default value is 16)
> 
> The threshold for seeing trouble appears to be '15' - ie it works with 14
> pcie-root-ports, fails with 15 pcie-root-ports .

Yep.  Each pci bridge gets 1k of io address space, we have 16k in total,
some of that is used for isa io and pci root bus, leaving enough io address
space for 14 pci bridges.

seabios assigns io address space to bridges only in case it finds a device
with io ports behind the bridge, or qemu asks for it via reserve-io hint.
The latter happens unconditionally now with acpi hotplug.

commit e2a6290aab578b2170c1f5909fa556385dc0d820
Author: Marcel Apfelbaum <marcel.apfelbaum>
Date:   Mon Aug 2 12:00:57 2021 +0300

    hw/pcie-root-port: Fix hotplug for PCI devices requiring IO
    
    Q35 has now ACPI hotplug enabled by default for PCI(e) devices.
    As opposed to native PCIe hotplug, guests like Fedora 34
    will not assign IO range to pcie-root-ports not supporting
    native hotplug, resulting into a regression.
    
    Reproduce by:
        qemu-bin -M q35 -device pcie-root-port,id=p1 -monitor stdio
        device_add e1000,bus=p1
    In the Guest OS the respective pcie-root-port will have the IO range
    disabled.
    
    Fix it by setting the "reserve-io" hint capability of the
    pcie-root-ports so the firmware will allocate the IO range instead.
    
    Acked-by: Igor Mammedov <imammedo>
    Signed-off-by: Marcel Apfelbaum <marcel>
    Message-Id: <20210802090057.1709775-1-marcel>
    Reviewed-by: Michael S. Tsirkin <mst>
    Signed-off-by: Michael S. Tsirkin <mst>

(In reply to Gerd Hoffmann from comment #9)
> (In reply to Daniel Berrangé from comment #6)
> > > - When setting /etc/nova/nova.conf libvirt num_pcie_ports 4, there's no issue. (default value is 16)
> > 
> > The threshold for seeing trouble appears to be '15' - ie it works with 14
> > pcie-root-ports, fails with 15 pcie-root-ports .
> 
> Yep.  Each pci bridge gets 1k of io address space, we have 16k in total,
> some of that is used for isa io and pci root bus, leaving enough io address
> space for 14 pci bridges.
> 
> seabios assigns io address space to bridges only in case it finds a device
> with io ports behind the bridge, or qemu asks for it via reserve-io hint.
> The latter happens unconditionally now with acpi hotplug.
> 
> commit e2a6290aab578b2170c1f5909fa556385dc0d820
> Author: Marcel Apfelbaum <marcel.apfelbaum>
> Date:   Mon Aug 2 12:00:57 2021 +0300
> 
>     hw/pcie-root-port: Fix hotplug for PCI devices requiring IO
>     
>     Q35 has now ACPI hotplug enabled by default for PCI(e) devices.
>     As opposed to native PCIe hotplug, guests like Fedora 34
>     will not assign IO range to pcie-root-ports not supporting
>     native hotplug, resulting into a regression.
>     
>     Reproduce by:
>         qemu-bin -M q35 -device pcie-root-port,id=p1 -monitor stdio
>         device_add e1000,bus=p1
>     In the Guest OS the respective pcie-root-port will have the IO range
>     disabled.
>     
>     Fix it by setting the "reserve-io" hint capability of the
>     pcie-root-ports so the firmware will allocate the IO range instead.
>     
>     Acked-by: Igor Mammedov <imammedo>
>     Signed-off-by: Marcel Apfelbaum <marcel>
>     Message-Id: <20210802090057.1709775-1-marcel>
>     Reviewed-by: Michael S. Tsirkin <mst>
>     Signed-off-by: Michael S. Tsirkin <mst>

We have a concern which we couldn't validate yet, but what if operators are adding PCI-PT devices. Is that counting toward this limitation? If an operator needs 20 PCI-PT, is it going to work?

(In reply to David Vallee Delisle from comment #10)
> We have a concern which we couldn't validate yet, but what if operators are
> adding PCI-PT devices. Is that counting toward this limitation? If an
> operator needs 20 PCI-PT, is it going to work?

The limit comes from the pcie-root-ports, and essentially every device needs its own pcie-root-port, so yes, PCI passthrough devices will count towards this limit.

QEMU needs to be fixed to get rid of this regression in consumption of I/O space.

Assigned to Amnon for initial triage per bz process and age of bug created or assigned to virt-maint without triage.

Is there a CS9 version of a patch/build from RHEL8 bug 2007129 ?

Hi David,

Can you please confirm the latest CentOS stream compose has fixed the issue? The fix should be already included in qemu-kvm-6.2.0-1.el9 and newer. Please close this bug if it works for you. Thank you.

Yes, it's fixed in the latest release.