Marcel - libvirt allows connecting a pcie-switch-upstream-port (x3130-upstream) directly into pcie-expander-bus (pxb-pcie), but this test indicates it doesn't work properly. I think I recall asking you if that was allowable, and wrote the code in libvirt to allow it based on your reply. So there are 3 possibilities: 1) I am remembering wrong, and libvirt shouldn't allow this, 2) there is a bug in one of the devices in qemu, or 3) something else unrelated is causing the crash. Which is it?

(In reply to Laine Stump from comment #2)
> Marcel - libvirt allows connecting a pcie-switch-upstream-port
> (x3130-upstream) directly into pcie-expander-bus (pxb-pcie), but this test
> indicates it doesn't work properly. I think I recall asking you if that was
> allowable, and wrote the code in libvirt to allow it based on your reply. So
> there are 3 possibilities: 1) I am remembering wrong, and libvirt shouldn't
> allow this, 2) there is a bug in one of the devices in qemu, or 3) something
> else unrelated is causing the crash. Which is it?

Hi Laine,
There is another possibility, that I didn't explain myself clear enough
and I am sorry if this is the case.
Regarding the problem, the PCIe root bus exposed by the pxb-pcie device
has the same rules as pcie.0 root bus.
Since you cannot add upstream ports directly into pcie.0 bus, you
cannot do that for the pxb-pcie bus either. (and this is why the kernel guest crashes)

You can argue we should prevent this configuration in QEMU, but there are
so many strange and not supported QEMU configurations that I thought
is better to enforce the sane ones in libvirt.

Thanks,
Marcel

Fix pushed upstream:

commit 10031fe5f218fe0acbf873a3063ce42a02fa83d9
Author: Laine Stump <laine>
Date:   Fri Aug 5 21:19:27 2016 -0400

    conf: improve error log when PCI devices don't match requested controller

Sorry, disregard Comment 4 - *this* is the upstream commit:

commit b70e54342bbd1756234e07ed6b22bdd3cd12b689
Author: Laine Stump <laine>
Date:   Thu Aug 4 13:04:12 2016 -0400

    conf: don't allow connecting upstream-port directly to pce-expander-bus

Verified on libvirt-2.0.0-5.el7.x86_64

define guest when attaching upstream port into pxb-pcie
<controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>

<controller type='pci' index='3' model='pcie-expander-bus'>
      <model name='pxb-pcie'/>
      <target busNr='10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='4' model='pcie-switch-upstream-port'>
      <model name='x3130-upstream'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='5' model='pcie-switch-downstream-port'>
      <model name='xio3130-downstream'/>
      <target chassis='5' port='0x0'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </controller>

# virsh define vm1-q35-pxb.xml
error: Failed to define domain from vm1-q35-pxb.xml
error: XML error: The device at PCI address 0000:03:00.0 cannot be plugged into the PCI controller with index='3'. It requires a controller that accepts a pci-switch-upstream-port.

If attaching root-port into pxb-pcie then attaching upstream-port, guest works

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2577.html