Bug 1363648 - Add support to attach dmi-to-pci-bridge (i82801b11-bridge) into pxb-pcie
Summary: Add support to attach dmi-to-pci-bridge (i82801b11-bridge) into pxb-pcie
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.3
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Laine Stump
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2016-08-03 09:31 UTC by Yang Yang
Modified: 2016-11-03 18:51 UTC (History)
4 users (show)

Fixed In Version: libvirt-2.0.0-5.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2016-11-03 18:51:31 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2016:2577 normal SHIPPED_LIVE Moderate: libvirt security, bug fix, and enhancement update 2016-11-03 12:07:06 UTC

Description Yang Yang 2016-08-03 09:31:05 UTC
Description of problem:
Libvirt does not support to attach dmi-to-pci-bridge (i82801b11-bridge) directly into pxb-pcie. From the POV of qemu, if it is supported by pcie.0, pxb-pcie should support it as well. So libvirt should support to attach dmi-to-pci-bridge (i82801b11-bridge) directly into pxb-pcie.

Version-Release number of selected component (if applicable):
libvirt-2.0.0-4.el7.x86_64
qemu-kvm-rhev-2.6.0-18.el7.x86_64


How reproducible:
100%

Steps to Reproduce:
1. create vm with pxb-pcie -- dmi-to-pci-bridge
<controller type='pci' index='0' model='pcie-root'/>
<controller type='pci' index='1' model='dmi-to-pci-bridge'/>

    <controller type='pci' index='2' model='pci-bridge'/>

    <controller type='pci' index='3' model='pcie-expander-bus'>
      <model name='pxb-pcie'/>
      <target busNr='10'/>
    </controller>

<controller type='pci' index='4' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x3' slot='0x0' function='0x0'/>
</controller>

# virsh create vm1-q35-pxb-tmp.xml
error: Failed to create domain from vm1-q35-pxb-tmp.xml
error: XML error: PCI bus is not compatible with the device at 0000:03:00.0. Device requires a PCI Express slot, which is not provided by bus 0000:03


Actual results:
Libvirt does not support attach dmi-to-pci-bridge into pxb-pcie
However, qemu supports that

Expected results:
Add support to attach dmi-to-pci-bridge into pxb-pcie


Additional info:

Comment 2 Laine Stump 2016-08-03 14:57:02 UTC
Marcel - is this actually supported by qemu?

Comment 3 Yang Yang 2016-08-04 01:58:15 UTC
Laine,

I attempted to attach dmi-to-pci-bridge (i82801b11-bridge) directly into pxb-pcie by qemu. Guest starts up normally 

e.g.

[root@rhel7_test ~]# /usr/libexec/qemu-kvm -machine q35,accel=kvm,usb=off \
> -smp 4,maxcpus=4,cores=2,threads=2,sockets=1 \
> -cpu SandyBridge,enforce \
> -m 2G \
> -boot strict=on \
> -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e \
> -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0 \
> -device pxb-pcie,id=pci.3,bus=pcie.0,bus_nr=10 \
> -device i82801b11-bridge,id=pci.4,bus=pci.3,addr=0x0 \
> -device pci-bridge,chassis_nr=5,id=pci.5,bus=pci.4,addr=0x0 \
> -drive file=/mnt/nfs2/RHEL-7.3-latest.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 \
> -device virtio-blk-pci,scsi=off,bus=pci.5,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
> -monitor stdio -spice port=5931,disable-ticketing -boot menu=on \
> -qmp tcp:0:6666,server,nowait

QEMU 2.6.0 monitor - type 'help' for more information
(qemu) info pci
  Bus 10, device   0, function 0:
    PCI bridge: PCI device 8086:244e
      BUS 10.
      secondary bus 11.
      subordinate bus 12.
      IO range [0xc000, 0xcfff]
      memory range [0xfe200000, 0xfe5fffff]
      prefetchable memory range [0xfd000000, 0xfd1fffff]
      id "pci.4"
  Bus 11, device   0, function 0:
    PCI bridge: PCI device 1b36:0001
      IRQ 10.
      BUS 11.
      secondary bus 12.
      subordinate bus 12.
      IO range [0xc000, 0xcfff]
      memory range [0xfe200000, 0xfe3fffff]
      prefetchable memory range [0xfd000000, 0xfd1fffff]
      BAR0: 64 bit memory at 0xfe400000 [0xfe4000ff].
      id "pci.5"
  Bus 12, device   3, function 0:
    SCSI controller: PCI device 1af4:1001
      IRQ 11.
      BAR0: I/O at 0xc000 [0xc03f].
      BAR1: 32 bit memory at 0xfe200000 [0xfe200fff].
      id "virtio-disk0"
  Bus  0, device   0, function 0:
    Host bridge: PCI device 8086:29c0
      id ""
  Bus  0, device   1, function 0:
    VGA controller: PCI device 1234:1111
      BAR0: 32 bit prefetchable memory at 0xfc000000 [0xfcffffff].
      BAR2: 32 bit memory at 0xfea70000 [0xfea70fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id ""
  Bus  0, device   2, function 0:
    Ethernet controller: PCI device 8086:100e
      IRQ 11.
      BAR0: 32 bit memory at 0xfea40000 [0xfea5ffff].
      BAR1: I/O at 0xe000 [0xe03f].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id ""
  Bus  0, device   3, function 0:
    Host bridge: PCI device 1b36:000b
      id "pci.3"
  Bus  0, device  30, function 0:
    PCI bridge: PCI device 8086:244e
      BUS 0.
      secondary bus 1.
      subordinate bus 2.
      IO range [0xd000, 0xdfff]
      memory range [0xfe600000, 0xfe9fffff]
      prefetchable memory range [0xfd200000, 0xfd3fffff]
      id "pci.1"
  Bus  1, device   0, function 0:
    PCI bridge: PCI device 1b36:0001
      IRQ 10.
      BUS 1.
      secondary bus 2.
      subordinate bus 2.
      IO range [0xd000, 0xdfff]
      memory range [0xfe600000, 0xfe7fffff]
      prefetchable memory range [0xfd200000, 0xfd3fffff]
      BAR0: 64 bit memory at 0xfe800000 [0xfe8000ff].
      id "pci.2"
  Bus  0, device  31, function 0:
    ISA bridge: PCI device 8086:2918
      id ""
  Bus  0, device  31, function 2:
    SATA controller: PCI device 8086:2922
      IRQ 10.
      BAR4: I/O at 0xe080 [0xe09f].
      BAR5: 32 bit memory at 0xfea71000 [0xfea71fff].
      id ""
  Bus  0, device  31, function 3:
    SMBus: PCI device 8086:2930
      IRQ 10.
      BAR4: I/O at 0x0700 [0x073f].
      id ""

Comment 4 Marcel Apfelbaum 2016-08-04 10:26:09 UTC
(In reply to Laine Stump from comment #2)
> Marcel - is this actually supported by qemu?

Yes, and if doesn't work is a bug.
The best way to think of pxb/pbx-pcie is that it exposes a new pcie.0 like bus.
The only difference is that you cannot have devices plugged directly into it (integrated device).

pxb has a built-in bridge
pxb-pcie - use pcie root ports.

Thanks,
Marcel

Comment 5 Laine Stump 2016-08-04 17:31:39 UTC
Since I'm needing to make a separate "connection class" for dmi-to-pci-bridge (previously I allowed a dmi-to-pci-bridge to be plugged into any place a standard PCIe endpoint device could be connected), I have another question - is it supported to connect a dmi-to-pci-bridge into 1) a pcie-root-port? 2) a pcie-switch-downstream-port? Or should I limit it to only pcie-root and pcie-expander-bus?

Comment 6 Laine Stump 2016-08-08 19:22:44 UTC
Marcel answered my questions in Comment 5 during an IRC conversation - the answer is that dmi-to-pci-bridge can be connected *only* to pcie-root or pcie-expander-bus, and nothing else.

Comment 7 Laine Stump 2016-08-10 15:07:42 UTC
Fix pushed upstream:

commit b70e3d0123fcb6e22e99d1b272239e03a84262cb
Author: Laine Stump <laine@laine.org>
Date:   Fri Aug 5 15:01:08 2016 -0400

    conf: restrict where dmi-to-pci-bridge can be connected

Comment 10 Yang Yang 2016-08-15 07:55:58 UTC
Verified on libvirt-2.0.0-5.el7.x86_64

Steps
1. attach dmi-to-pci-bridge to pxb-pcie  
<controller type='pci' index='0' model='pcie-root'>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-expander-bus'>
      <target busNr='100'>
        <node>1</node>
      </target>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='4' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='5' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </controller>

    <interface type='direct'>
      <mac address='52:54:00:65:18:d8'/>
      <source dev='eno1' mode='bridge'/>
      <target dev='macvtap0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x01' function='0x0'/>
    </interface>

#virsh define vm1-q35-Sec.xml
#virsh start vm1-q35-Sec

#ps -ef|grep qemu
-device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e
-device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0
-device pxb-pcie,bus_nr=100,id=pci.3,numa_node=1,bus=pcie.0,addr=0x2
-device i82801b11-bridge,id=pci.4,bus=pci.3,addr=0x0
-device pci-bridge,chassis_nr=5,id=pci.5,bus=pci.4,addr=0x0
-netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=28
-device virtionet-pci,netdev=hostnet0,id=net0,mac=52:54:00:65:18:d8,bus=pci.5,addr=0x1

checked in guest, nic is usable

2. attach dmi-to-pci-bridge to dmi-to-pci-bridge
 <controller type='pci' index='0' model='pcie-root'>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>

# virsh define vm1-q35-sec.xml
error: Failed to define domain from vm1-q35-sec.xml
error: XML error: The device at PCI address 0000:01:01.0 cannot be plugged into the PCI controller with index='1'. It requires a controller that accepts a dmi-to-pci-bridge.
	
3.attach dmi-to-pci-bridge to pci-bridge
<controller type='pci' index='0' model='pcie-root'>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x1f' function='0x0'/>
    </controller>

# virsh define vm1-q35-sec.xml
error: Failed to define domain from vm1-q35-sec.xml
error: XML error: The device at PCI address 0000:02:1f.0 cannot be plugged into the PCI controller with index='2'. It requires a controller that accepts a dmi-to-pci-bridge.

4. attach dmi-to-pci-bridge to pcie-root-port
 <controller type='pci' index='0' model='pcie-root'>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'/>
    <controller type='pci' index='4' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x0' function='0x0'/>
    </controller>

# virsh define vm1-q35-sec.xml
error: Failed to define domain from vm1-q35-sec.xml
error: XML error: The device at PCI address 0000:03:00.0 cannot be plugged into the PCI controller with index='3'. It requires a controller that accepts a dmi-to-pci-bridge.

5. attach dmi-to-pci-bridge to pcie-switch-upstream-port
 <controller type='pci' index='0' model='pcie-root'>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'/>
    <controller type='pci' index='4' model='pcie-switch-upstream-port'/>
    <controller type='pci' index='5' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x0' function='0x0'/>
    </controller>

# virsh define vm1-q35-sec.xml
error: Failed to define domain from vm1-q35-sec.xml
error: XML error: The device at PCI address 0000:04:00.0 cannot be plugged into the PCI controller with index='4'. It requires a controller that accepts a dmi-to-pci-bridge.

6. attach dmi-to-pci-bridge to pcie-switch-downstream-port
<controller type='pci' index='0' model='pcie-root'>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'/>
    <controller type='pci' index='4' model='pcie-switch-upstream-port'/>
    <controller type='pci' index='5' model='pcie-switch-downstream-port'/>
    <controller type='pci' index='6' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x0' function='0x0'/>
    </controller>

# virsh define vm1-q35-sec.xml
error: Failed to define domain from vm1-q35-sec.xml
error: XML error: The device at PCI address 0000:05:00.0 cannot be plugged into the PCI controller with index='5'. It requires a controller that accepts a dmi-to-pci-bridge.

The results are expected.

Comment 12 errata-xmlrpc 2016-11-03 18:51:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2577.html


Note You need to log in before you can comment on or make changes to this bug.