Bug 1520821

I started working on adding pcie-pci-bridge support to libvirt.
A few questions:

* Is the bridge supposed to be plugged straight into pcie.0 like
  i82801b11-bridge, or into a PCIe Root Port?
* Can it be hotplugged?
* How many slots does the bridge have? How are they numbered?
* Can PCI devices be hotplugged in those slots?

That should be all I need for now.

Hi Andrea,

(In reply to Andrea Bolognani from comment #2)
> I started working on adding pcie-pci-bridge support to libvirt.

Great news!

> A few questions:
> 
> * Is the bridge supposed to be plugged straight into pcie.0 like
>   i82801b11-bridge, or into a PCIe Root Port?

Into a PCIe Root Port, is a PCI express device.

> * Can it be hotplugged?

Yes, since is a PCI Express device you can hot-plug it,
in order to do so you need to:

1. Reserve at least an extra bus number on the Root Port you are hot-plugging it into:
   -device pcie-root-port,id=p1,bus-reserve=1 -device pcie-pci-bridge,bus=p1
2. Upgrade the host to use the latest Linux Kernel, I think 4.14 would do, but 4.15 RC is a better bet.

> * How many slots does the bridge have?

On the other "side" is a regular PCI bridge, so 32.
 
> How are they numbered?

Similar to the PCI bridge.

> * Can PCI devices be hotplugged in those slots?

Yes, but be aware we are not using APCI based hot plug, but SHPC based
hot plug meaning:
1. Slot 0 cannot be used (SHPC sits there)
2. The hot plug will not work with ancient guests which do not support
SHPC hot plug.

> 
> That should be all I need for now.

Be sure you read docs/pcie_pci_bridge.txt on QEMU's git
for a quick overview.

Good luck!
Marcel

(In reply to Marcel Apfelbaum from comment #3)
> > A few questions:
> > 
> > * Is the bridge supposed to be plugged straight into pcie.0 like
> >   i82801b11-bridge, or into a PCIe Root Port?
> 
> Into a PCIe Root Port, is a PCI express device.

Okay, so instead of

  pcie.0 -> i82801b11-bridge -> pci-bridge -> endpoint

we'll have

  pcie.0 -> pcie-root-port -> pcie-pci-bridge -> endpoint

That's good, it means we're not using any additional bus numbers
compared to the previous solution :)

> > * Can it be hotplugged?
> 
> Yes, since is a PCI Express device you can hot-plug it,
> in order to do so you need to:
> 
> 1. Reserve at least an extra bus number on the Root Port you are
> hot-plugging it into:
>    -device pcie-root-port,id=p1,bus-reserve=1 -device pcie-pci-bridge,bus=p1
> 2. Upgrade the host to use the latest Linux Kernel, I think 4.14 would do,
> but 4.15 RC is a better bet.

From the document you mentioned, it seems like only SeaBIOS
supports the bus-reserve option at the moment. This implies the
bridge can be hotplugged with (some) Q35 guests, but not with
mach-virt guests, where EDKII is used instead.

Moreover, even if we limited ourselves to Q35 guests using SeaBIOS,
we would still need to have one PCIe Root Port configured
specifically so that pcie-pci-bridge hotplug is possible. If the
user has to get so much out of their way to prepare for traditional
PCI hotplug, why not just go the extra mile and configure the
pcie-pci-bridge right away?

All things considered, it seems to me like hotplug is probably more
trouble than it's worth, so I'll probably go ahead and not implement
it at all. We can revisit this decision later on, of course.

Patches posted upstream:

  https://www.redhat.com/archives/libvir-list/2018-March/msg01739.html

Pushed upstream.

commit 898edc961763c4f81e25aaf7bd81017829dd697a
Author: Andrea Bolognani <abologna>
Date:   Wed Mar 28 12:48:21 2018 +0200

    conf: Prefer pcie-to-pci-bridge to dmi-to-pci-bridge
    
    Both pcie-to-pci-bridge and dmi-to-pci-bridge can be used to
    create a traditional PCI topology in a pure PCIe guest such as
    those using the x86_64/q35 or aarch64/virt machine type;
    however, the former should be preferred, as it doesn't need to
    obey limitation of real hardware and is completely
    architecture-agnostic.
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1520821
    
    Signed-off-by: Andrea Bolognani <abologna>
    Reviewed-by: John Ferlan <jferlan>

v4.2.0-58-g898edc961

Hi Andrea,

In /docs/formatdomain.html.in, it said: libvirt will never auto-assign a PCI device to a PCIe slot, it will allow manual specification of such an assignment.

I test two scenarios according to this, but I'm not sure if it's expected, can you help to check with it? Thanks in advance.

Version:
libvirt-4.5.0-3.el7.x86_64

Scenario 1: Edit guest xml with multiple pcie-to-pci-bridge.
# virsh edit q35
...
<controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </controller>
<controller type='pci' index='2' model='pcie-to-pci-bridge'/>
<controller type='pci' index='3' model='pcie-to-pci-bridge'/>
...
error: internal error: Cannot automatically add a new PCI bus for a device with connect flags 800
Failed. Try again? [y,n,i,f,?]:

--(1)Only with index='2', it can edit successfully.
--(2)If the results is expected, I think we should have a more clear error info.


Scenario 2: Define and start guest with multiple pcie-to-pci-bridge.
# vi q35.xml
...
<controller type='pci' index='1' model='pcie-root-port'>
<controller type='pci' index='2' model='pcie-to-pci-pridge'/>
<controller type='pci' index='3' model='pcie-to-pci-bridge'/>
...
# virsh define q35.xml
# virsh start q35
# virsh dumpxml test | grep pcie-to-pci-bridge -a5

-----No output, pcie-to-pci-bridge controller can't be added to guest successfully. If the results is expected, maybe we should have some error info.

(In reply to Meina Li from comment #12)
> Hi Andrea,
> 
> In /docs/formatdomain.html.in, it said: libvirt will never auto-assign a PCI
> device to a PCIe slot, it will allow manual specification of such an
> assignment.
> 
> I test two scenarios according to this, but I'm not sure if it's expected,
> can you help to check with it? Thanks in advance.

Neither of the scenarios outlined below are related to that
specific bit of the documentation, since you're not manually
specifying the PCI address for any device...

> Scenario 1: Edit guest xml with multiple pcie-to-pci-bridge.
> # virsh edit q35
> ...
> <controller type='pci' index='0' model='pcie-root'/>
>     <controller type='pci' index='1' model='pcie-root-port'>
>       <model name='pcie-root-port'/>
>       <target chassis='1' port='0x8'/>
>       <address type='pci' domain='0x0000' bus='0x00' slot='0x01'
> function='0x0'/>
>     </controller>
> <controller type='pci' index='2' model='pcie-to-pci-bridge'/>
> <controller type='pci' index='3' model='pcie-to-pci-bridge'/>
> ...
> error: internal error: Cannot automatically add a new PCI bus for a device
> with connect flags 800
> Failed. Try again? [y,n,i,f,?]:
> 
> --(1)Only with index='2', it can edit successfully.
> --(2)If the results is expected, I think we should have a more clear error
> info.

Each pcie-to-pci-bridge needs a pcie-root-port to plug into, and
since you only have one pcie-root-port in your configuration the
second pcie-to-pci-bridge can't find a suitable slot to plug into.

libvirt in general doesn't cope extremely well with having only a
partial PCI topology as input, so as a user you're really supposed
to either provide just the endpoints or the full PCI topology -
anything else will probably result in an error.

I agree that it would be nice to either figure out a solution
automatically or at least report a better error in this kind of
situation, but that's something that's really out of scope for
this specific bug and honestly probably not very high in the list
of priorities in general.

> Scenario 2: Define and start guest with multiple pcie-to-pci-bridge.
> # vi q35.xml
> ...
> <controller type='pci' index='1' model='pcie-root-port'>
> <controller type='pci' index='2' model='pcie-to-pci-pridge'/>
> <controller type='pci' index='3' model='pcie-to-pci-bridge'/>
> ...
> # virsh define q35.xml

I would expect this to fail with the same error as above. Can
you please share the full XML definition?

> # virsh start q35
> # virsh dumpxml test | grep pcie-to-pci-bridge -a5

Looks like you're running start and dumpxml on different guests
here.

Hi, 

Thanks for your reply.

That's my mistake and guest can't be defined in scenario 2. It will fail with error:
Error：internal error: Cannot automatically add a new PCI bus for a device with connect flags 800

This error isn't very clear and it really need to report a better error in this kind of situation. I'll file a bug with low priority to record it.

(In reply to Meina Li from comment #14)
> Hi, 
> 
> Thanks for your reply.
> 
> That's my mistake and guest can't be defined in scenario 2. It will fail
> with error:
> Error：internal error: Cannot automatically add a new PCI bus for a device
> with connect flags 800

Okay, that makes more sense :)

> This error isn't very clear and it really need to report a better error in
> this kind of situation. I'll file a bug with low priority to record it.

Feel free to do that; just don't hold your breath waiting for it
to be fixed...

Hi Andrea,

Whether we no longer use pcie-to-pci-pridge both in qemu and libvirt? If we also don't want to use it in libvirt, we should change this bug's status and related info in https://libvirt.org/formatdomain.html.

I found that it will not support pcie-to-pci-pridge from qemu-kvm-rhev-2.12.0-9.el7.x86_64:
error: unsupported configuration: The 'pcie-pci-bridge' device is not supported by this QEMU binary

(In reply to Meina Li from comment #16)
> Hi Andrea,
> 
> Whether we no longer use pcie-to-pci-pridge both in qemu and libvirt? If we
> also don't want to use it in libvirt, we should change this bug's status and
> related info in https://libvirt.org/formatdomain.html.
> 
> I found that it will not support pcie-to-pci-pridge from
> qemu-kvm-rhev-2.12.0-9.el7.x86_64:
> error: unsupported configuration: The 'pcie-pci-bridge' device is not
> supported by this QEMU binary

It looks like it was decided not to support pcie-to-pci-bridge
downstream, see

  https://bugzilla.redhat.com/show_bug.cgi?id=1390329#c6

Since QEMU doesn't include the device, obviously libvirt won't be
able to use it either; with that in mind, it makes sense to close
this bug as WONTFIX too. I'll do that now.

This is, however, a downstream decision and won't affect upstream
status at all, so no changes are needed there.

The decision to disable the device in QEMU has been reverted[1],
so moving the bug back to ON_QA.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1390329#c15

Verified version:

qemu-kvm-rhev-2.12.0-11.el7.x86_64
libvirt-4.5.0-7.el7.x86_64

Test Scenarios:

Scenario 1: Start a q35 guest with single or multiple pcie-to-pci-bridge controllers.

Scenario 2: Attributes supported in pcie-to-ci-bridge controller.

Scenario 3: Hotplug/Hot-unplug pcie-to-pci-bridge itself.

Scenario 4: Hotplug/Hot-unplug pci devices to pcie-to-pci-bridge and check status in guest.

Scenario 5: The number of available slots for pcie-to-pci-bridge controller. 

Detailed test steps will be described at the following comment after the test.

Following comment 19, test on seabios-q35 guest and ovmf-q35 guest respectively.

Scenario 1: Start a q35 guest with single or multiple pcie-to-pci-bridge controllers.
1. With single pcie-to-pci-bridge controller.
# virsh dumpxml lmn
…
 <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='3' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
 <sound model='ich6'>
      <alias name='sound0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x01' function='0x0'/>
    </sound>
…
# ps aux | grep qemu | sed 's/-device/\n-device/g'
...
-device pcie-root-port,port=0x10,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2 
-device pcie-root-port,port=0x11,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1 
-device pcie-pci-bridge,id=pci.3,bus=pci.1,addr=0x0 
-device intel-hda,id=sound0,bus=pci.3,addr=0x1
…

2. With multiple pcie-to-ci-bridge controller.
# virsh edit lmn
...
<controller type='pci' index='1' model='pcie-root-port'/>
<controller type='pci' index='2' model='pcie-root-port'/>
<controller type='pci' index='3' model='pcie-root-port'/>
<controller type='pci' index='4' model='pcie-to-pci-bridge'/>
<controller type='pci' index='5' model='pcie-to-pci-bridge'/>
<controller type='pci' index='6' model='pcie-to-pci-bridge'/>
…
# virsh start lmn
# virsh dumpxml lmn | grep pcie-to-pci-bridge -a4
<controller type='pci' index='4' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='5' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='6' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
3. Only with pci device(rtl8139 nic) and without pcie-to-pci-bridge controller.
---passed, a pcie-to-pci-bridge controller will automatically be added

Scenario 2: Attributes supported in pcie-to-ci-bridge controller
1. Invalid index(Index range should be 1~255.
1) Define a q35 VM, the index of pcie-to-ci-bridge is 0.
<controller type='pci' index='0' model=pcie-to-ci-bridge/>
# virsh define /tmp/q35.xml
error: Failed to define domain from /tmp/q35.xml
error: XML error: Multiple 'pci' controllers with index '0'
2)  Define a q35 VM, the index of pcie-to-ci-bridge is 256.
<controller type='pci' index='256' model=pcie-to-ci-bridge/>
# virsh define /tmp/q35.xml
error: Failed to define domain from /tmp/q35.xml
error: internal error: a PCI slot is needed to connect a PCI controller model='pcie-root-port', but none is available, and it cannot be automatically added
3) Define a q35 VM, the index of pcie-to-ci-bridge is abc.
<controller type='pci' index='abc' model=pcie-to-ci-bridge/>
# virsh define /tmp/q35.xml
error: Failed to define domain from /tmp/q35.xml
error: internal error: Cannot parse controller index abc

2. With invalid model name.
<model name='pci-bridge'/>
# virsh define /tmp/q35.xml
error: Failed to define domain from /tmp/q35.xml
error: unsupported configuration: Option 'modelName' has invalid value for PCI controller with index '2', model 'pcie-to-pci-bridge' and modelName 'pci-bridge'

3. With invalid address(not plug into pcie-root-port controller). 
 <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
# virsh define /tmp/q35.xml
error: Failed to define domain from /tmp/q35.xml
error: XML error: The device at PCI address 0000:00:0a.0 cannot be plugged into the PCI controller with index='0'. It requires a controller that accepts a pcie-to-pci-bridge.

4. With index <=bus
 <controller type='pci' index='2' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
# virsh define /tmp/q35.xml
error: Failed to define domain from /tmp/q35.xml
error: XML error: The device at PCI address 0000:03:00.0 cannot be plugged into the PCI controller with index='3'. It requires a controller that accepts a pcie-to-pci-bridge.

Following comment 19, test on seabios-q35 guest and ovmf-q35 guest respectively.

Scenario 3: Hotplug/Hot-unplug pcie-to-pci-bridge itself.

1. Start a guest without pcie-to-pci-bridge in guest and only with pcie-root-port.
# virsh dumpxml lmn | grep pcie-root-port -a5
<controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
2. Prepare a xml of pcie-to-pci-bridge controller.
# cat pcie-to-pci-bridge.xml
    <controller type='pci' index='2' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
    </controller>
3. Hotplug the pcie-to-pci-bridge itself
# virsh attach-device lmn pcie-to-pci-bridge.xml 
error: Failed to attach device from pcie-to-pci-bridge.xml
error: Operation not supported: 'pci' controller cannot be hot plugged.

--- Libvirt doesn't support the hotplug of this controller. And actually don't find bus-reserver in libvirt code.

Scenario 4: Hotplug/Hot-unplug pci devices to pcie-to-pci-bridge and check status in guest.

4.1.1 Prepare a running guest with pcie-to-pci-bridge controller (has another pci device in the tested pcie-to-pci-bridge).
# virsh dumpxml lmn | grep pcie-to-pci-bridge -a5
<controller type='pci' index='2' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
 <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </sound>

4.1.2 Prepare an rtl8139/e1000 interface.
# cat interface.xml
<interface type='network'>
      <mac address='52:54:00:44:a4:e5'/>
      <source network='default'/>
      <model type='e1000'/>
    </interface>

4.1.3 Attach interface device to guest and check interface.
# virsh attach-device lmn interface.xml 
Device attached successfully
# virsh dumpxml lmn | grep /interface -B7
    <interface type='network'>
      <mac address='52:54:00:44:a4:e4'/>
      <source network='default' bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:44:a4:e5'/>
      <source network='default' bridge='virbr0'/>
      <target dev='vnet1'/>
      <model type='e1000'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>

4.1.4 Login guest and check network.
(guest)# ip a
…
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:44:a4:e4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.59/24 brd 192.168.122.255 scope global noprefixroute dynamic eth0
       valid_lft 3472sec preferred_lft 3472sec
    inet6 fe80::5054:ff:fe44:a4e4/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
3: ens1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:44:a4:e5 brd ff:ff:ff:ff:ff:ff

4.1.5 Detach interface.
# virsh detach-device lmn interface.xml 
Device detached successfully

4.2.1. Prepare a running guest with pcie-to-pci-bridge controller (no other pci device in the tested pcie-to-pci-bridge).
# virsh dumpxml lmn | grep pcie-to-pci-bridge -a5
<controller type='pci' index='2' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>

4.2.2. Attach interface device to guest and check interface.
# virsh attach-device lmn interface.xml 
Device attached successfully
# virsh dumpxml lmn | grep /interface -B7
    <interface type='network'>
      <mac address='52:54:00:44:a4:e4'/>
      <source network='default' bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:44:a4:e5'/>
      <source network='default' bridge='virbr0'/>
      <target dev='vnet1'/>
      <model type='e1000'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>

4.2.3. Login guest and check network.
(guest)# ip a
...
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:44:a4:e4 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.59/24 brd 192.168.122.255 scope global noprefixroute dynamic eth0
       valid_lft 3485sec preferred_lft 3485sec
    inet6 fe80::5054:ff:fe44:a4e4/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

--- For seabios+q35, can't get pci device in guest and the bug 1619884 has already tracked it.
--- For ovmf+q35, can get pci device in guest.

Following comment 19, test on seabios-q35 guest and ovmf-q35 guest respectively.
1. Prepare a script.
# cat bridge-num.xml 
#/bin/bash
DOM=lmn
for i in {1..32};do
    echo "---$i---"
    virsh attach-interface $DOM network default --model rtl8139
done
echo the slots of pci-bridge is $(virsh dumpxml $DOM|grep "bus='0x02'.*function='0x0'"|wc -l)

2. Run the script.
# sh bridge-num.xml
…
---31---
Interface attached successfully

---32---
error: Failed to attach interface
error: internal error: No more available PCI slots

the slots of pci-bridge is 31

According to comment 20 to comment 22, move this bug to be verified.

There is no difference from libvirt's point of view, and we have no way of configuring it. I also know 0 of the details of either method. My advice would be to not expect hotplug to work correctly during guest boot, then you'll always be pleased with the results.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2018:3113