Bug 1358712 - pci-expander-bus should only connect to pci-root, and pcie-expander-bus should only connect to pcie-root
Summary: pci-expander-bus should only connect to pci-root, and pcie-expander-bus shoul...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.3
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Laine Stump
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-07-21 10:23 UTC by Yang Yang
Modified: 2016-11-03 18:50 UTC (History)
4 users (show)

Fixed In Version: libvirt-2.0.0-5.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-11-03 18:50:47 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2016:2577 normal SHIPPED_LIVE Moderate: libvirt security, bug fix, and enhancement update 2016-11-03 12:07:06 UTC

Description Yang Yang 2016-07-21 10:23:32 UTC
Description of problem:
I have 2 pxb controllers in domain xml. 1st pxb controller is attached
to pci root bus. 2nd pxb controller is attached to 1st pxb. VM cannot
startup normally and paused. 

Addressed qemu log here
qemu-kvm: -device pxb,bus_nr=98,id=pci.2,bus=pci.1,addr=0x0: PXB devices
can be attached only to root bus.

So libvirt should reject to attach pxb controller to both pxb and pci bridge

Version-Release number of selected component (if applicable):
libvirt-2.0.0-2.el7.x86_64
qemu-kvm-rhev-2.6.0-13.el7.x86_64

How reproducible:
100%

Steps to Reproduce:
1.create vm with following xml
<controller type='pci' index='1' model='pci-expander-bus'>
      <model name='pxb'/>
      <target busNr='100'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </controller>

<controller type='pci' index='2' model='pci-expander-bus'>
      <model name='pxb'/>
      <target busNr='98'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x1' function='0x0'/>
    </controller>

# virsh create vm1-pxb.xml

# virsh list
 Id    Name                           State
----------------------------------------------------
 16    vm1-pxb                        paused

2.
3.

Actual results:
Guest cannot startup normally when pxb is connected to etheir pxb or pci bridge

Expected results:
Reject to attach pxb controller to both pxb and pci bridge

Additional info:
2016-07-21T10:07:54.504115Z qemu-kvm: -device pxb,bus_nr=98,id=pci.2,bus=pci.1,addr=0x1: PXB devices can be attached only to root bus.

Comment 2 Yang Yang 2016-08-05 03:24:29 UTC
pxb-pcie controller has the similar problem. Libvirt should limit pxb-pcie only to pcie-root. I attempted negative configuration as following. Although qemu provides error, I still hope libvirt disallow those incorrect configurations

e.g.
1. attach pxb-pcie to dmi-to-pci-bridge
<controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
 <controller type='pci' index='3' model='pcie-expander-bus'>
      <model name='pxb-pcie'/>
      <target busNr='10'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x1' function='0x0'/>    </controller>

# virsh create vm1-q35-pxb.xml
error: Failed to create domain from vm1-q35-pxb.xml
error: internal error: qemu unexpectedly closed the monitor: 2016-07-28T03:23:39.910935Z qemu-kvm: -device pxb-pcie,bus_nr=10,id=pci.3,bus=pci.1,addr=0x1: pxb-pcie devices cannot reside on a PCI bus!
2016-07-28T03:23:39.910976Z qemu-kvm: -device pxb-pcie,bus_nr=10,id=pci.3,bus=pci.1,addr=0x1: Device initialization failed

1. attach pxb-pcie to dmi-to-pci-bridge
<controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
 <controller type='pci' index='3' model='pcie-expander-bus'>
      <model name='pxb-pcie'/>
      <target busNr='10'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x1' function='0x0'/>    </controller>

# virsh create vm1-q35-pxb.xml
error: Failed to create domain from vm1-q35-pxb.xml
error: internal error: qemu unexpectedly closed the monitor: 2016-07-28T03:23:39.910935Z qemu-kvm: -device pxb-pcie,bus_nr=10,id=pci.3,bus=pci.1,addr=0x1: pxb-pcie devices cannot reside on a PCI bus!
2016-07-28T03:23:39.910976Z qemu-kvm: -device pxb-pcie,bus_nr=10,id=pci.3,bus=pci.1,addr=0x1: Device initialization failed

2. attach pxb-pcie to pci bridge
<controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
 <controller type='pci' index='3' model='pcie-expander-bus'>
      <model name='pxb-pcie'/>
      <target busNr='10'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x1' function='0x0'/>    </controller>

# virsh create vm1-q35-pxb.xml
error: Failed to create domain from vm1-q35-pxb.xml
error: internal error: qemu unexpectedly closed the monitor: 2016-07-28T03:25:28.984589Z qemu-kvm: -device pxb-pcie,bus_nr=10,id=pci.3,bus=pci.2,addr=0x1: pxb-pcie devices cannot reside on a PCI bus!
2016-07-28T03:25:28.984629Z qemu-kvm: -device pxb-pcie,bus_nr=10,id=pci.3,bus=pci.2,addr=0x1: Device initialization failed

3. attach pxb-pcie to pxb-pcie
<controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
<controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
 <controller type='pci' index='3' model='pcie-expander-bus'>
      <model name='pxb-pcie'/>
      <target busNr='10'/>
    </controller>
 <controller type='pci' index='4' model='pcie-expander-bus'>
      <model name='pxb-pcie'/>
      <target busNr='12'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>

# virsh create vm1-q35-pxb.xml
error: Failed to create domain from vm1-q35-pxb.xml
error: XML error: PCI bus is not compatible with the device at 0000:03:00.0. Device requires a PCI Express slot, which is not provided by bus 0000:03

4. attach pxb-pcie to pcie-root-port
<controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>

 <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
<controller type='pci' index='3' model='pcie-root-port'/>
 <controller type='pci' index='4' model='pcie-expander-bus'>
      <model name='pxb-pcie'/>
      <target busNr='12'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>

# virsh create vm1-q35-pxb.xml
error: Failed to create domain from vm1-q35-pxb.xml
error: internal error: process exited while connecting to monitor: 2016-07-28T03:32:40.499077Z qemu-kvm: -device pxb-pcie,bus_nr=12,id=pci.4,bus=pci.3,addr=0x0: PXB devices can be attached only to root bus.
2016-07-28T03:32:40.499126Z qemu-kvm: -device pxb-pcie,bus_nr=12,id=pci.4,bus=pci.3,addr=0x0: Device initialization failed

5. attach pxb-pcie to upstream port
<controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
 <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
<controller type='pci' index='3' model='pcie-root-port'/>
<controller type='pci' index='4' model='pcie-switch-upstream-port'/>
 <controller type='pci' index='5' model='pcie-expander-bus'>
      <model name='pxb-pcie'/>
      <target busNr='12'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </controller>

# virsh create vm1-q35-pxb.xml
error: Failed to create domain from vm1-q35-pxb.xml
error: XML error: PCI bus is not compatible with the device at 0000:04:00.0. Device requires a PCI Express slot, which is not provided by bus 0000:04


	
6. attach pxb-pcie to downstream port
<controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
<controller type='pci' index='3' model='pcie-root-port'/>
<controller type='pci' index='4' model='pcie-switch-upstream-port'/>
<controller type='pci' index='5' model='pcie-switch-downstream-port'/>
 <controller type='pci' index='6' model='pcie-expander-bus'>
<model name='pxb-pcie'/>
      <target busNr='12'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </controller>

# virsh create vm1-q35-pxb.xml
error: Failed to create domain from vm1-q35-pxb.xml
error: internal error: qemu unexpectedly closed the monitor: 2016-07-28T03:35:40.962342Z qemu-kvm: -device pxb-pcie,bus_nr=12,id=pci.6,bus=pci.5,addr=0x0: PXB devices can be attached only to root bus.
2016-07-28T03:35:40.962391Z qemu-kvm: -device pxb-pcie,bus_nr=12,id=pci.6,bus=pci.5,addr=0x0: Device initialization failed

Comment 3 Laine Stump 2016-08-06 02:48:41 UTC
Marcel,

Before adding these restrictions, I just want to verify that:

1) a pxb is not allowed to be plugged into pci-bridge or another pxb (i.e. *only* into the pci-root aka pci.0)

2) a pxb-pcie is not allowed to be plugged into anything except directly into pcie.0.

Are both of these statements correct?

Comment 4 Marcel Apfelbaum 2016-08-06 10:29:54 UTC
(In reply to Laine Stump from comment #3)
> Marcel,
> 
> Before adding these restrictions, I just want to verify that:
> 
> 1) a pxb is not allowed to be plugged into pci-bridge or another pxb (i.e.
> *only* into the pci-root aka pci.0)
> 

Correct.

> 2) a pxb-pcie is not allowed to be plugged into anything except directly
> into pcie.0.
> 

Correct again.

> Are both of these statements correct?

Yes, by design.

Thanks,
Marcel

Comment 5 Laine Stump 2016-08-10 15:09:12 UTC
Fix pushed upstream:

commit a220f43a65cca6c6f2ca268cdbbf8f997b2e2b13
Author: Laine Stump <laine@laine.org>
Date:   Sat Aug 6 19:03:31 2016 -0400

    conf: restrict expander buses to connect only to a root bus

Comment 8 Yang Yang 2016-08-12 06:09:13 UTC
Verified on libvirt-2.0.0-5.el7.x86_64

For pxb

1. attach pxb to pxb
<controller type='pci' index='1' model='pci-expander-bus'>
      <model name='pxb'/>
      <target busNr='100'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </controller>

<controller type='pci' index='2' model='pci-expander-bus'>
      <model name='pxb'/>
      <target busNr='98'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x1' function='0x0'/>
    </controller>

# virsh define vm1-pxb.xml
error: Failed to define domain from vm1-pxb.xml
error: XML error: The device at PCI address 0000:01:01.0 cannot be plugged into the PCI controller with index='1'. It requires a controller that accepts a pci-expander-bus.

2. attach pxb to pci bridge
<controller type='pci' index='1' model='pci-expander-bus'>
      <model name='pxb'/>
      <target busNr='100'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </controller>
<controller type='pci' index='2' model='pci-bridge'/>
<controller type='pci' index='3' model='pci-expander-bus'>
      <model name='pxb'/>
      <target busNr='98'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x1' function='0x0'/>
    </controller>

# virsh define vm1-pxb.xml
error: Failed to define domain from vm1-pxb.xml
error: XML error: The device at PCI address 0000:02:00.0 cannot be plugged into the PCI controller with index='2'. It requires a controller that accepts a pci-expander-bus.

For pxb-pcie
 1. attach pxb-pcie to dmi-to-pci-bridge
<controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
 <controller type='pci' index='3' model='pcie-expander-bus'>
      <model name='pxb-pcie'/>
      <target busNr='10'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x1' function='0x0'/>   
</controller>

# virsh define vm1-q35-pxb.xml
error: Failed to define domain from vm1-q35-pxb.xml
error: XML error: The device at PCI address 0000:01:01.0 cannot be plugged into the PCI controller with index='1'. It requires a controller that accepts a pcie-expander-bus.

2. attach pxb-pcie to pci bridge
<controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
 <controller type='pci' index='3' model='pcie-expander-bus'>
      <model name='pxb-pcie'/>
      <target busNr='10'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x1' function='0x0'/>    </controller>

# virsh define vm1-q35-pxb.xml
error: Failed to define domain from vm1-q35-pxb.xml
error: XML error: The device at PCI address 0000:02:01.0 cannot be plugged into the PCI controller with index='2'. It requires a controller that accepts a pcie-expander-bus.

3. attach pxb-pcie to pxb-pcie
<controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
<controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
 <controller type='pci' index='3' model='pcie-expander-bus'>
      <model name='pxb-pcie'/>
      <target busNr='10'/>
    </controller>
 <controller type='pci' index='4' model='pcie-expander-bus'>
      <model name='pxb-pcie'/>
      <target busNr='12'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>

# virsh define vm1-q35-pxb.xml
error: Failed to define domain from vm1-q35-pxb.xml
error: XML error: The device at PCI address 0000:03:00.0 cannot be plugged into the PCI controller with index='3'. It requires a controller that accepts a pcie-expander-bus.

4. attach pxb-pcie to pcie-root-port
<controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>

 <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
<controller type='pci' index='3' model='pcie-root-port'/>
 <controller type='pci' index='4' model='pcie-expander-bus'>
      <model name='pxb-pcie'/>
      <target busNr='12'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>

# virsh define vm1-q35-pxb.xml
error: Failed to define domain from vm1-q35-pxb.xml
error: XML error: The device at PCI address 0000:03:00.0 cannot be plugged into the PCI controller with index='3'. It requires a controller that accepts a pcie-expander-bus.

5. attach pxb-pcie to upstream port
<controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
 <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
<controller type='pci' index='3' model='pcie-root-port'/>
<controller type='pci' index='4' model='pcie-switch-upstream-port'/>
 <controller type='pci' index='5' model='pcie-expander-bus'>
      <model name='pxb-pcie'/>
      <target busNr='12'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </controller>

# virsh define vm1-q35-pxb.xml
error: Failed to define domain from vm1-q35-pxb.xml
error: XML error: The device at PCI address 0000:04:00.0 cannot be plugged into the PCI controller with index='4'. It requires a controller that accepts a pcie-expander-bus.

6. attach pxb-pcie to downstream port
<controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
<controller type='pci' index='3' model='pcie-root-port'/>
<controller type='pci' index='4' model='pcie-switch-upstream-port'/>
<controller type='pci' index='5' model='pcie-switch-downstream-port'/>
 <controller type='pci' index='6' model='pcie-expander-bus'>
<model name='pxb-pcie'/>
      <target busNr='12'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </controller>

# virsh define vm1-q35-pxb.xml
error: Failed to define domain from vm1-q35-pxb.xml
error: XML error: The device at PCI address 0000:05:00.0 cannot be plugged into the PCI controller with index='5'. It requires a controller that accepts a pcie-expander-bus.

For the negative testing, I got the proper error.
pxb works well when attaching to pci-root as well as pxb-pcie when attaching to pcie-root

Comment 10 errata-xmlrpc 2016-11-03 18:50:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2577.html


Note You need to log in before you can comment on or make changes to this bug.