Bug 1358712
| Summary: | pci-expander-bus should only connect to pci-root, and pcie-expander-bus should only connect to pcie-root | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Yang Yang <yanyang> |
| Component: | libvirt | Assignee: | Laine Stump <laine> |
| Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 7.3 | CC: | dyuan, hhan, marcel, rbalakri |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | libvirt-2.0.0-5.el7 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2016-11-03 18:50:47 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
pxb-pcie controller has the similar problem. Libvirt should limit pxb-pcie only to pcie-root. I attempted negative configuration as following. Although qemu provides error, I still hope libvirt disallow those incorrect configurations
e.g.
1. attach pxb-pcie to dmi-to-pci-bridge
<controller type='pci' index='0' model='pcie-root'>
<alias name='pcie.0'/>
</controller>
<controller type='pci' index='1' model='dmi-to-pci-bridge'>
<model name='i82801b11-bridge'/>
<alias name='pci.1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
</controller>
<controller type='pci' index='2' model='pci-bridge'>
<model name='pci-bridge'/>
<target chassisNr='2'/>
<alias name='pci.2'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='3' model='pcie-expander-bus'>
<model name='pxb-pcie'/>
<target busNr='10'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x1' function='0x0'/> </controller>
# virsh create vm1-q35-pxb.xml
error: Failed to create domain from vm1-q35-pxb.xml
error: internal error: qemu unexpectedly closed the monitor: 2016-07-28T03:23:39.910935Z qemu-kvm: -device pxb-pcie,bus_nr=10,id=pci.3,bus=pci.1,addr=0x1: pxb-pcie devices cannot reside on a PCI bus!
2016-07-28T03:23:39.910976Z qemu-kvm: -device pxb-pcie,bus_nr=10,id=pci.3,bus=pci.1,addr=0x1: Device initialization failed
1. attach pxb-pcie to dmi-to-pci-bridge
<controller type='pci' index='0' model='pcie-root'>
<alias name='pcie.0'/>
</controller>
<controller type='pci' index='1' model='dmi-to-pci-bridge'>
<model name='i82801b11-bridge'/>
<alias name='pci.1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
</controller>
<controller type='pci' index='2' model='pci-bridge'>
<model name='pci-bridge'/>
<target chassisNr='2'/>
<alias name='pci.2'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='3' model='pcie-expander-bus'>
<model name='pxb-pcie'/>
<target busNr='10'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x1' function='0x0'/> </controller>
# virsh create vm1-q35-pxb.xml
error: Failed to create domain from vm1-q35-pxb.xml
error: internal error: qemu unexpectedly closed the monitor: 2016-07-28T03:23:39.910935Z qemu-kvm: -device pxb-pcie,bus_nr=10,id=pci.3,bus=pci.1,addr=0x1: pxb-pcie devices cannot reside on a PCI bus!
2016-07-28T03:23:39.910976Z qemu-kvm: -device pxb-pcie,bus_nr=10,id=pci.3,bus=pci.1,addr=0x1: Device initialization failed
2. attach pxb-pcie to pci bridge
<controller type='pci' index='0' model='pcie-root'>
<alias name='pcie.0'/>
</controller>
<controller type='pci' index='1' model='dmi-to-pci-bridge'>
<model name='i82801b11-bridge'/>
<alias name='pci.1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
</controller>
<controller type='pci' index='2' model='pci-bridge'>
<model name='pci-bridge'/>
<target chassisNr='2'/>
<alias name='pci.2'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='3' model='pcie-expander-bus'>
<model name='pxb-pcie'/>
<target busNr='10'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x1' function='0x0'/> </controller>
# virsh create vm1-q35-pxb.xml
error: Failed to create domain from vm1-q35-pxb.xml
error: internal error: qemu unexpectedly closed the monitor: 2016-07-28T03:25:28.984589Z qemu-kvm: -device pxb-pcie,bus_nr=10,id=pci.3,bus=pci.2,addr=0x1: pxb-pcie devices cannot reside on a PCI bus!
2016-07-28T03:25:28.984629Z qemu-kvm: -device pxb-pcie,bus_nr=10,id=pci.3,bus=pci.2,addr=0x1: Device initialization failed
3. attach pxb-pcie to pxb-pcie
<controller type='pci' index='0' model='pcie-root'>
<alias name='pcie.0'/>
</controller>
<controller type='pci' index='1' model='dmi-to-pci-bridge'>
<model name='i82801b11-bridge'/>
<alias name='pci.1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
</controller>
<controller type='pci' index='2' model='pci-bridge'>
<model name='pci-bridge'/>
<target chassisNr='2'/>
<alias name='pci.2'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='3' model='pcie-expander-bus'>
<model name='pxb-pcie'/>
<target busNr='10'/>
</controller>
<controller type='pci' index='4' model='pcie-expander-bus'>
<model name='pxb-pcie'/>
<target busNr='12'/>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
</controller>
# virsh create vm1-q35-pxb.xml
error: Failed to create domain from vm1-q35-pxb.xml
error: XML error: PCI bus is not compatible with the device at 0000:03:00.0. Device requires a PCI Express slot, which is not provided by bus 0000:03
4. attach pxb-pcie to pcie-root-port
<controller type='pci' index='0' model='pcie-root'>
<alias name='pcie.0'/>
</controller>
<controller type='pci' index='1' model='dmi-to-pci-bridge'>
<model name='i82801b11-bridge'/>
<alias name='pci.1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
</controller>
<controller type='pci' index='2' model='pci-bridge'>
<model name='pci-bridge'/>
<target chassisNr='2'/>
<alias name='pci.2'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'/>
<controller type='pci' index='4' model='pcie-expander-bus'>
<model name='pxb-pcie'/>
<target busNr='12'/>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
</controller>
# virsh create vm1-q35-pxb.xml
error: Failed to create domain from vm1-q35-pxb.xml
error: internal error: process exited while connecting to monitor: 2016-07-28T03:32:40.499077Z qemu-kvm: -device pxb-pcie,bus_nr=12,id=pci.4,bus=pci.3,addr=0x0: PXB devices can be attached only to root bus.
2016-07-28T03:32:40.499126Z qemu-kvm: -device pxb-pcie,bus_nr=12,id=pci.4,bus=pci.3,addr=0x0: Device initialization failed
5. attach pxb-pcie to upstream port
<controller type='pci' index='0' model='pcie-root'>
<alias name='pcie.0'/>
</controller>
<controller type='pci' index='1' model='dmi-to-pci-bridge'>
<model name='i82801b11-bridge'/>
<alias name='pci.1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
</controller>
<controller type='pci' index='2' model='pci-bridge'>
<model name='pci-bridge'/>
<target chassisNr='2'/>
<alias name='pci.2'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'/>
<controller type='pci' index='4' model='pcie-switch-upstream-port'/>
<controller type='pci' index='5' model='pcie-expander-bus'>
<model name='pxb-pcie'/>
<target busNr='12'/>
<address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
</controller>
# virsh create vm1-q35-pxb.xml
error: Failed to create domain from vm1-q35-pxb.xml
error: XML error: PCI bus is not compatible with the device at 0000:04:00.0. Device requires a PCI Express slot, which is not provided by bus 0000:04
6. attach pxb-pcie to downstream port
<controller type='pci' index='0' model='pcie-root'>
<alias name='pcie.0'/>
</controller>
<controller type='pci' index='1' model='dmi-to-pci-bridge'>
<model name='i82801b11-bridge'/>
<alias name='pci.1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
</controller>
<controller type='pci' index='2' model='pci-bridge'>
<model name='pci-bridge'/>
<target chassisNr='2'/>
<alias name='pci.2'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'/>
<controller type='pci' index='4' model='pcie-switch-upstream-port'/>
<controller type='pci' index='5' model='pcie-switch-downstream-port'/>
<controller type='pci' index='6' model='pcie-expander-bus'>
<model name='pxb-pcie'/>
<target busNr='12'/>
<address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
</controller>
# virsh create vm1-q35-pxb.xml
error: Failed to create domain from vm1-q35-pxb.xml
error: internal error: qemu unexpectedly closed the monitor: 2016-07-28T03:35:40.962342Z qemu-kvm: -device pxb-pcie,bus_nr=12,id=pci.6,bus=pci.5,addr=0x0: PXB devices can be attached only to root bus.
2016-07-28T03:35:40.962391Z qemu-kvm: -device pxb-pcie,bus_nr=12,id=pci.6,bus=pci.5,addr=0x0: Device initialization failed
Marcel, Before adding these restrictions, I just want to verify that: 1) a pxb is not allowed to be plugged into pci-bridge or another pxb (i.e. *only* into the pci-root aka pci.0) 2) a pxb-pcie is not allowed to be plugged into anything except directly into pcie.0. Are both of these statements correct? (In reply to Laine Stump from comment #3) > Marcel, > > Before adding these restrictions, I just want to verify that: > > 1) a pxb is not allowed to be plugged into pci-bridge or another pxb (i.e. > *only* into the pci-root aka pci.0) > Correct. > 2) a pxb-pcie is not allowed to be plugged into anything except directly > into pcie.0. > Correct again. > Are both of these statements correct? Yes, by design. Thanks, Marcel Fix pushed upstream:
commit a220f43a65cca6c6f2ca268cdbbf8f997b2e2b13
Author: Laine Stump <laine>
Date: Sat Aug 6 19:03:31 2016 -0400
conf: restrict expander buses to connect only to a root bus
Verified on libvirt-2.0.0-5.el7.x86_64
For pxb
1. attach pxb to pxb
<controller type='pci' index='1' model='pci-expander-bus'>
<model name='pxb'/>
<target busNr='100'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</controller>
<controller type='pci' index='2' model='pci-expander-bus'>
<model name='pxb'/>
<target busNr='98'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x1' function='0x0'/>
</controller>
# virsh define vm1-pxb.xml
error: Failed to define domain from vm1-pxb.xml
error: XML error: The device at PCI address 0000:01:01.0 cannot be plugged into the PCI controller with index='1'. It requires a controller that accepts a pci-expander-bus.
2. attach pxb to pci bridge
<controller type='pci' index='1' model='pci-expander-bus'>
<model name='pxb'/>
<target busNr='100'/>
<alias name='pci.1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</controller>
<controller type='pci' index='2' model='pci-bridge'/>
<controller type='pci' index='3' model='pci-expander-bus'>
<model name='pxb'/>
<target busNr='98'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x1' function='0x0'/>
</controller>
# virsh define vm1-pxb.xml
error: Failed to define domain from vm1-pxb.xml
error: XML error: The device at PCI address 0000:02:00.0 cannot be plugged into the PCI controller with index='2'. It requires a controller that accepts a pci-expander-bus.
For pxb-pcie
1. attach pxb-pcie to dmi-to-pci-bridge
<controller type='pci' index='0' model='pcie-root'>
<alias name='pcie.0'/>
</controller>
<controller type='pci' index='1' model='dmi-to-pci-bridge'>
<model name='i82801b11-bridge'/>
<alias name='pci.1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
</controller>
<controller type='pci' index='2' model='pci-bridge'>
<model name='pci-bridge'/>
<target chassisNr='2'/>
<alias name='pci.2'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='3' model='pcie-expander-bus'>
<model name='pxb-pcie'/>
<target busNr='10'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x1' function='0x0'/>
</controller>
# virsh define vm1-q35-pxb.xml
error: Failed to define domain from vm1-q35-pxb.xml
error: XML error: The device at PCI address 0000:01:01.0 cannot be plugged into the PCI controller with index='1'. It requires a controller that accepts a pcie-expander-bus.
2. attach pxb-pcie to pci bridge
<controller type='pci' index='0' model='pcie-root'>
<alias name='pcie.0'/>
</controller>
<controller type='pci' index='1' model='dmi-to-pci-bridge'>
<model name='i82801b11-bridge'/>
<alias name='pci.1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
</controller>
<controller type='pci' index='2' model='pci-bridge'>
<model name='pci-bridge'/>
<target chassisNr='2'/>
<alias name='pci.2'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='3' model='pcie-expander-bus'>
<model name='pxb-pcie'/>
<target busNr='10'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x1' function='0x0'/> </controller>
# virsh define vm1-q35-pxb.xml
error: Failed to define domain from vm1-q35-pxb.xml
error: XML error: The device at PCI address 0000:02:01.0 cannot be plugged into the PCI controller with index='2'. It requires a controller that accepts a pcie-expander-bus.
3. attach pxb-pcie to pxb-pcie
<controller type='pci' index='0' model='pcie-root'>
<alias name='pcie.0'/>
</controller>
<controller type='pci' index='1' model='dmi-to-pci-bridge'>
<model name='i82801b11-bridge'/>
<alias name='pci.1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
</controller>
<controller type='pci' index='2' model='pci-bridge'>
<model name='pci-bridge'/>
<target chassisNr='2'/>
<alias name='pci.2'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='3' model='pcie-expander-bus'>
<model name='pxb-pcie'/>
<target busNr='10'/>
</controller>
<controller type='pci' index='4' model='pcie-expander-bus'>
<model name='pxb-pcie'/>
<target busNr='12'/>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
</controller>
# virsh define vm1-q35-pxb.xml
error: Failed to define domain from vm1-q35-pxb.xml
error: XML error: The device at PCI address 0000:03:00.0 cannot be plugged into the PCI controller with index='3'. It requires a controller that accepts a pcie-expander-bus.
4. attach pxb-pcie to pcie-root-port
<controller type='pci' index='0' model='pcie-root'>
<alias name='pcie.0'/>
</controller>
<controller type='pci' index='1' model='dmi-to-pci-bridge'>
<model name='i82801b11-bridge'/>
<alias name='pci.1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
</controller>
<controller type='pci' index='2' model='pci-bridge'>
<model name='pci-bridge'/>
<target chassisNr='2'/>
<alias name='pci.2'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'/>
<controller type='pci' index='4' model='pcie-expander-bus'>
<model name='pxb-pcie'/>
<target busNr='12'/>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
</controller>
# virsh define vm1-q35-pxb.xml
error: Failed to define domain from vm1-q35-pxb.xml
error: XML error: The device at PCI address 0000:03:00.0 cannot be plugged into the PCI controller with index='3'. It requires a controller that accepts a pcie-expander-bus.
5. attach pxb-pcie to upstream port
<controller type='pci' index='0' model='pcie-root'>
<alias name='pcie.0'/>
</controller>
<controller type='pci' index='1' model='dmi-to-pci-bridge'>
<model name='i82801b11-bridge'/>
<alias name='pci.1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
</controller>
<controller type='pci' index='2' model='pci-bridge'>
<model name='pci-bridge'/>
<target chassisNr='2'/>
<alias name='pci.2'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'/>
<controller type='pci' index='4' model='pcie-switch-upstream-port'/>
<controller type='pci' index='5' model='pcie-expander-bus'>
<model name='pxb-pcie'/>
<target busNr='12'/>
<address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
</controller>
# virsh define vm1-q35-pxb.xml
error: Failed to define domain from vm1-q35-pxb.xml
error: XML error: The device at PCI address 0000:04:00.0 cannot be plugged into the PCI controller with index='4'. It requires a controller that accepts a pcie-expander-bus.
6. attach pxb-pcie to downstream port
<controller type='pci' index='0' model='pcie-root'>
<alias name='pcie.0'/>
</controller>
<controller type='pci' index='1' model='dmi-to-pci-bridge'>
<model name='i82801b11-bridge'/>
<alias name='pci.1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
</controller>
<controller type='pci' index='2' model='pci-bridge'>
<model name='pci-bridge'/>
<target chassisNr='2'/>
<alias name='pci.2'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</controller>
<controller type='pci' index='3' model='pcie-root-port'/>
<controller type='pci' index='4' model='pcie-switch-upstream-port'/>
<controller type='pci' index='5' model='pcie-switch-downstream-port'/>
<controller type='pci' index='6' model='pcie-expander-bus'>
<model name='pxb-pcie'/>
<target busNr='12'/>
<address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
</controller>
# virsh define vm1-q35-pxb.xml
error: Failed to define domain from vm1-q35-pxb.xml
error: XML error: The device at PCI address 0000:05:00.0 cannot be plugged into the PCI controller with index='5'. It requires a controller that accepts a pcie-expander-bus.
For the negative testing, I got the proper error.
pxb works well when attaching to pci-root as well as pxb-pcie when attaching to pcie-root
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2016-2577.html |
Description of problem: I have 2 pxb controllers in domain xml. 1st pxb controller is attached to pci root bus. 2nd pxb controller is attached to 1st pxb. VM cannot startup normally and paused. Addressed qemu log here qemu-kvm: -device pxb,bus_nr=98,id=pci.2,bus=pci.1,addr=0x0: PXB devices can be attached only to root bus. So libvirt should reject to attach pxb controller to both pxb and pci bridge Version-Release number of selected component (if applicable): libvirt-2.0.0-2.el7.x86_64 qemu-kvm-rhev-2.6.0-13.el7.x86_64 How reproducible: 100% Steps to Reproduce: 1.create vm with following xml <controller type='pci' index='1' model='pci-expander-bus'> <model name='pxb'/> <target busNr='100'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-expander-bus'> <model name='pxb'/> <target busNr='98'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x1' function='0x0'/> </controller> # virsh create vm1-pxb.xml # virsh list Id Name State ---------------------------------------------------- 16 vm1-pxb paused 2. 3. Actual results: Guest cannot startup normally when pxb is connected to etheir pxb or pci bridge Expected results: Reject to attach pxb controller to both pxb and pci bridge Additional info: 2016-07-21T10:07:54.504115Z qemu-kvm: -device pxb,bus_nr=98,id=pci.2,bus=pci.1,addr=0x1: PXB devices can be attached only to root bus.