1363648 – Add support to attach dmi-to-pci-bridge (i82801b11-bridge) into pxb-pcie

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1363648 - Add support to attach dmi-to-pci-bridge (i82801b11-bridge) into pxb-pcie

Summary: Add support to attach dmi-to-pci-bridge (i82801b11-bridge) into pxb-pcie

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	libvirt
Sub Component:
Version:	7.3
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	unspecified
Target Milestone:	rc
Target Release:	---
Assignee:	Laine Stump
QA Contact:	Virtualization Bugs
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2016-08-03 09:31 UTC by Yang Yang
Modified:	2016-11-03 18:51 UTC (History)
CC List:	4 users (show)
Fixed In Version:	libvirt-2.0.0-5.el7
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2016-11-03 18:51:31 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHSA-2016:2577	0	normal	SHIPPED_LIVE	Moderate: libvirt security, bug fix, and enhancement update	2016-11-03 12:07:06 UTC

Description Yang Yang 2016-08-03 09:31:05 UTC

Description of problem:
Libvirt does not support to attach dmi-to-pci-bridge (i82801b11-bridge) directly into pxb-pcie. From the POV of qemu, if it is supported by pcie.0, pxb-pcie should support it as well. So libvirt should support to attach dmi-to-pci-bridge (i82801b11-bridge) directly into pxb-pcie.

Version-Release number of selected component (if applicable):
libvirt-2.0.0-4.el7.x86_64
qemu-kvm-rhev-2.6.0-18.el7.x86_64


How reproducible:
100%

Steps to Reproduce:
1. create vm with pxb-pcie -- dmi-to-pci-bridge
<controller type='pci' index='0' model='pcie-root'/>
<controller type='pci' index='1' model='dmi-to-pci-bridge'/>

    <controller type='pci' index='2' model='pci-bridge'/>

    <controller type='pci' index='3' model='pcie-expander-bus'>
      <model name='pxb-pcie'/>
      <target busNr='10'/>
    </controller>

<controller type='pci' index='4' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x3' slot='0x0' function='0x0'/>
</controller>

# virsh create vm1-q35-pxb-tmp.xml
error: Failed to create domain from vm1-q35-pxb-tmp.xml
error: XML error: PCI bus is not compatible with the device at 0000:03:00.0. Device requires a PCI Express slot, which is not provided by bus 0000:03


Actual results:
Libvirt does not support attach dmi-to-pci-bridge into pxb-pcie
However, qemu supports that

Expected results:
Add support to attach dmi-to-pci-bridge into pxb-pcie


Additional info:

Comment 2 Laine Stump 2016-08-03 14:57:02 UTC

Marcel - is this actually supported by qemu?

Comment 3 Yang Yang 2016-08-04 01:58:15 UTC

Laine,

I attempted to attach dmi-to-pci-bridge (i82801b11-bridge) directly into pxb-pcie by qemu. Guest starts up normally 

e.g.

[root@rhel7_test ~]# /usr/libexec/qemu-kvm -machine q35,accel=kvm,usb=off \
> -smp 4,maxcpus=4,cores=2,threads=2,sockets=1 \
> -cpu SandyBridge,enforce \
> -m 2G \
> -boot strict=on \
> -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e \
> -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0 \
> -device pxb-pcie,id=pci.3,bus=pcie.0,bus_nr=10 \
> -device i82801b11-bridge,id=pci.4,bus=pci.3,addr=0x0 \
> -device pci-bridge,chassis_nr=5,id=pci.5,bus=pci.4,addr=0x0 \
> -drive file=/mnt/nfs2/RHEL-7.3-latest.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 \
> -device virtio-blk-pci,scsi=off,bus=pci.5,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \
> -monitor stdio -spice port=5931,disable-ticketing -boot menu=on \
> -qmp tcp:0:6666,server,nowait

QEMU 2.6.0 monitor - type 'help' for more information
(qemu) info pci
  Bus 10, device   0, function 0:
    PCI bridge: PCI device 8086:244e
      BUS 10.
      secondary bus 11.
      subordinate bus 12.
      IO range [0xc000, 0xcfff]
      memory range [0xfe200000, 0xfe5fffff]
      prefetchable memory range [0xfd000000, 0xfd1fffff]
      id "pci.4"
  Bus 11, device   0, function 0:
    PCI bridge: PCI device 1b36:0001
      IRQ 10.
      BUS 11.
      secondary bus 12.
      subordinate bus 12.
      IO range [0xc000, 0xcfff]
      memory range [0xfe200000, 0xfe3fffff]
      prefetchable memory range [0xfd000000, 0xfd1fffff]
      BAR0: 64 bit memory at 0xfe400000 [0xfe4000ff].
      id "pci.5"
  Bus 12, device   3, function 0:
    SCSI controller: PCI device 1af4:1001
      IRQ 11.
      BAR0: I/O at 0xc000 [0xc03f].
      BAR1: 32 bit memory at 0xfe200000 [0xfe200fff].
      id "virtio-disk0"
  Bus  0, device   0, function 0:
    Host bridge: PCI device 8086:29c0
      id ""
  Bus  0, device   1, function 0:
    VGA controller: PCI device 1234:1111
      BAR0: 32 bit prefetchable memory at 0xfc000000 [0xfcffffff].
      BAR2: 32 bit memory at 0xfea70000 [0xfea70fff].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0000fffe].
      id ""
  Bus  0, device   2, function 0:
    Ethernet controller: PCI device 8086:100e
      IRQ 11.
      BAR0: 32 bit memory at 0xfea40000 [0xfea5ffff].
      BAR1: I/O at 0xe000 [0xe03f].
      BAR6: 32 bit memory at 0xffffffffffffffff [0x0003fffe].
      id ""
  Bus  0, device   3, function 0:
    Host bridge: PCI device 1b36:000b
      id "pci.3"
  Bus  0, device  30, function 0:
    PCI bridge: PCI device 8086:244e
      BUS 0.
      secondary bus 1.
      subordinate bus 2.
      IO range [0xd000, 0xdfff]
      memory range [0xfe600000, 0xfe9fffff]
      prefetchable memory range [0xfd200000, 0xfd3fffff]
      id "pci.1"
  Bus  1, device   0, function 0:
    PCI bridge: PCI device 1b36:0001
      IRQ 10.
      BUS 1.
      secondary bus 2.
      subordinate bus 2.
      IO range [0xd000, 0xdfff]
      memory range [0xfe600000, 0xfe7fffff]
      prefetchable memory range [0xfd200000, 0xfd3fffff]
      BAR0: 64 bit memory at 0xfe800000 [0xfe8000ff].
      id "pci.2"
  Bus  0, device  31, function 0:
    ISA bridge: PCI device 8086:2918
      id ""
  Bus  0, device  31, function 2:
    SATA controller: PCI device 8086:2922
      IRQ 10.
      BAR4: I/O at 0xe080 [0xe09f].
      BAR5: 32 bit memory at 0xfea71000 [0xfea71fff].
      id ""
  Bus  0, device  31, function 3:
    SMBus: PCI device 8086:2930
      IRQ 10.
      BAR4: I/O at 0x0700 [0x073f].
      id ""

Comment 4 Marcel Apfelbaum 2016-08-04 10:26:09 UTC

(In reply to Laine Stump from comment #2)
> Marcel - is this actually supported by qemu?

Yes, and if doesn't work is a bug.
The best way to think of pxb/pbx-pcie is that it exposes a new pcie.0 like bus.
The only difference is that you cannot have devices plugged directly into it (integrated device).

pxb has a built-in bridge
pxb-pcie - use pcie root ports.

Thanks,
Marcel

Comment 5 Laine Stump 2016-08-04 17:31:39 UTC

Since I'm needing to make a separate "connection class" for dmi-to-pci-bridge (previously I allowed a dmi-to-pci-bridge to be plugged into any place a standard PCIe endpoint device could be connected), I have another question - is it supported to connect a dmi-to-pci-bridge into 1) a pcie-root-port? 2) a pcie-switch-downstream-port? Or should I limit it to only pcie-root and pcie-expander-bus?

Comment 6 Laine Stump 2016-08-08 19:22:44 UTC

Marcel answered my questions in Comment 5 during an IRC conversation - the answer is that dmi-to-pci-bridge can be connected *only* to pcie-root or pcie-expander-bus, and nothing else.

Comment 7 Laine Stump 2016-08-10 15:07:42 UTC

Fix pushed upstream:

commit b70e3d0123fcb6e22e99d1b272239e03a84262cb
Author: Laine Stump <laine>
Date:   Fri Aug 5 15:01:08 2016 -0400

    conf: restrict where dmi-to-pci-bridge can be connected

Comment 10 Yang Yang 2016-08-15 07:55:58 UTC

Verified on libvirt-2.0.0-5.el7.x86_64

Steps
1. attach dmi-to-pci-bridge to pxb-pcie  
<controller type='pci' index='0' model='pcie-root'>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-expander-bus'>
      <target busNr='100'>
        <node>1</node>
      </target>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='4' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='5' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </controller>

    <interface type='direct'>
      <mac address='52:54:00:65:18:d8'/>
      <source dev='eno1' mode='bridge'/>
      <target dev='macvtap0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x01' function='0x0'/>
    </interface>

#virsh define vm1-q35-Sec.xml
#virsh start vm1-q35-Sec

#ps -ef|grep qemu
-device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e
-device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x0
-device pxb-pcie,bus_nr=100,id=pci.3,numa_node=1,bus=pcie.0,addr=0x2
-device i82801b11-bridge,id=pci.4,bus=pci.3,addr=0x0
-device pci-bridge,chassis_nr=5,id=pci.5,bus=pci.4,addr=0x0
-netdev tap,fd=26,id=hostnet0,vhost=on,vhostfd=28
-device virtionet-pci,netdev=hostnet0,id=net0,mac=52:54:00:65:18:d8,bus=pci.5,addr=0x1

checked in guest, nic is usable

2. attach dmi-to-pci-bridge to dmi-to-pci-bridge
 <controller type='pci' index='0' model='pcie-root'>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>

# virsh define vm1-q35-sec.xml
error: Failed to define domain from vm1-q35-sec.xml
error: XML error: The device at PCI address 0000:01:01.0 cannot be plugged into the PCI controller with index='1'. It requires a controller that accepts a dmi-to-pci-bridge.
	
3.attach dmi-to-pci-bridge to pci-bridge
<controller type='pci' index='0' model='pcie-root'>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x1f' function='0x0'/>
    </controller>

# virsh define vm1-q35-sec.xml
error: Failed to define domain from vm1-q35-sec.xml
error: XML error: The device at PCI address 0000:02:1f.0 cannot be plugged into the PCI controller with index='2'. It requires a controller that accepts a dmi-to-pci-bridge.

4. attach dmi-to-pci-bridge to pcie-root-port
 <controller type='pci' index='0' model='pcie-root'>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'/>
    <controller type='pci' index='4' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x0' function='0x0'/>
    </controller>

# virsh define vm1-q35-sec.xml
error: Failed to define domain from vm1-q35-sec.xml
error: XML error: The device at PCI address 0000:03:00.0 cannot be plugged into the PCI controller with index='3'. It requires a controller that accepts a dmi-to-pci-bridge.

5. attach dmi-to-pci-bridge to pcie-switch-upstream-port
 <controller type='pci' index='0' model='pcie-root'>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'/>
    <controller type='pci' index='4' model='pcie-switch-upstream-port'/>
    <controller type='pci' index='5' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x0' function='0x0'/>
    </controller>

# virsh define vm1-q35-sec.xml
error: Failed to define domain from vm1-q35-sec.xml
error: XML error: The device at PCI address 0000:04:00.0 cannot be plugged into the PCI controller with index='4'. It requires a controller that accepts a dmi-to-pci-bridge.

6. attach dmi-to-pci-bridge to pcie-switch-downstream-port
<controller type='pci' index='0' model='pcie-root'>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'/>
    <controller type='pci' index='4' model='pcie-switch-upstream-port'/>
    <controller type='pci' index='5' model='pcie-switch-downstream-port'/>
    <controller type='pci' index='6' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x0' function='0x0'/>
    </controller>

# virsh define vm1-q35-sec.xml
error: Failed to define domain from vm1-q35-sec.xml
error: XML error: The device at PCI address 0000:05:00.0 cannot be plugged into the PCI controller with index='5'. It requires a controller that accepts a dmi-to-pci-bridge.

The results are expected.

Comment 12 errata-xmlrpc 2016-11-03 18:51:31 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHSA-2016-2577.html

Note You need to log in before you can comment on or make changes to this bug.