Bug 1389369

Summary: libvirt mistakenly allows only slot 0 to be used on pcie-expander-bus
Product: Red Hat Enterprise Linux 7 Reporter: Jaroslav Reznik <jreznik>
Component: libvirtAssignee: Laine Stump <laine>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: unspecified Docs Contact:
Priority: high    
Version: 7.3CC: berrange, dyuan, laine, lmanasko, marcel, rbalakri, sherold, xuzhang, yanyang
Target Milestone: rcKeywords: ZStream
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-2.0.0-10.el7_3.1 Doc Type: Bug Fix
Doc Text:
Previously, the libvirt library allowed only use of slot 0 on the pcie-expander-bus controllers, even though slots 0 - 31 could be used. The restriction has been removed and slots 0 - 31 now can be used.
 Story Points: ---
Clone Of: 1382079 Environment:
Last Closed: 2016-12-06 17:11:22 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1382079    
Bug Blocks:    

Description Jaroslav Reznik 2016-10-27 13:42:48 UTC 
This bug has been copied from bug #1382079 and has been proposed
to be backported to 7.3 z-stream (EUS).
Comment 6 Yang Yang 2016-11-08 09:26:08 UTC 
Verified on libvirt-2.0.0-10.el7_3.1.x86_64

1. positive test:
attach 1 dmi-to-pci-bridge controller and 1 pcie-root-port controller to pcie-expander-bus

<controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pcie-expander-bus'>
      <model name='pxb-pcie'/>
      <target busNr='100'>
        <node>1</node>
      </target>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='ioh3420'/>
      <target chassis='4' port='0x0'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='5' model='pcie-switch-upstream-port'>
      <model name='x3130-upstream'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='6' model='pcie-switch-downstream-port'>
      <model name='xio3130-downstream'/>
      <target chassis='6' port='0x0'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='7' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <alias name='pci.7'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x1f' function='0x0'/>
    </controller>
    <controller type='pci' index='8' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='8'/>
      <alias name='pci.8'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x03' function='0x0'/>
    </controller>
    <interface type='direct'>
      <mac address='52:54:00:e9:03:50'/>
      <source dev='eno1' mode='bridge'/>
      <target dev='macvtap0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:b0:8d:2b'/>
      <source network='default' bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x03' function='0x0'/>
    </interface>

1.1. check nics in guest, make sure both nics have ip address
guest]# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.66.5.66  netmask 255.255.252.0  broadcast 10.66.7.255
        inet6 fe80::5054:ff:fee9:350  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:e9:03:50  txqueuelen 1000  (Ethernet)
        RX packets 138  bytes 17576 (17.1 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 27  bytes 5204 (5.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.122.240  netmask 255.255.255.0  broadcast 192.168.122.255
        inet6 fe80::5054:ff:feb0:8d2b  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:b0:8d:2b  txqueuelen 1000  (Ethernet)
        RX packets 77  bytes 6402 (6.2 KiB)
        RX errors 0  dropped 6  overruns 0  frame 0
        TX packets 49  bytes 5602 (5.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

1.2. check controller in guest
guest]#lspci
00:02.0 Host bridge: Red Hat, Inc. Device 000b
	Subsystem: Red Hat, Inc Device 1100
	Flags: 66MHz, fast devsel

64:00.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 0 (rev 02) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 11, NUMA node 1
	Bus: primary=64, secondary=65, subordinate=67, sec-latency=0
	Memory behind bridge: fe800000-fe9fffff
	Prefetchable memory behind bridge: 00000000fc200000-00000000fc3fffff
	Capabilities: [90] Express Root Port (Slot+), MSI 00
	Capabilities: [60] MSI: Enable- Count=1/2 Maskable+ 64bit-
	Capabilities: [40] Subsystem: Intel Corporation Device 0000
	Capabilities: [100] Advanced Error Reporting
	Kernel driver in use: pcieport
	Kernel modules: shpchp

64:1f.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 92) (prog-if 01 [Subtractive decode])
	Flags: bus master, 66MHz, fast devsel, latency 0, NUMA node 1
	Bus: primary=64, secondary=68, subordinate=69, sec-latency=0
	I/O behind bridge: 0000c000-0000cfff
	Memory behind bridge: fe000000-fe3fffff
	Prefetchable memory behind bridge: 00000000fc000000-00000000fc1fffff
	Capabilities: [50] Subsystem: Device 0000:0000

65:00.0 PCI bridge: Texas Instruments XIO3130 PCI Express Switch (Upstream) (rev 02) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, NUMA node 1
	Bus: primary=65, secondary=66, subordinate=67, sec-latency=0
	Memory behind bridge: fe800000-fe9fffff
	Prefetchable memory behind bridge: 00000000fc200000-00000000fc3fffff
	Capabilities: [90] Express Upstream Port, MSI 00
	Capabilities: [80] Subsystem: Device 0000:0000
	Capabilities: [70] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [100] Advanced Error Reporting
	Kernel driver in use: pcieport
	Kernel modules: shpchp

66:00.0 PCI bridge: Texas Instruments XIO3130 PCI Express Switch (Downstream) (rev 01) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, NUMA node 1
	Bus: primary=66, secondary=67, subordinate=67, sec-latency=0
	Memory behind bridge: fe800000-fe9fffff
	Prefetchable memory behind bridge: 00000000fc200000-00000000fc3fffff
	Capabilities: [90] Express Downstream Port (Slot+), MSI 00
	Capabilities: [80] Subsystem: Device 0000:0000
	Capabilities: [70] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [100] Advanced Error Reporting
	Kernel driver in use: pcieport
	Kernel modules: shpchp

67:00.0 Ethernet controller: Red Hat, Inc Virtio network device (rev 01)
	Subsystem: Red Hat, Inc Device 1100
	Flags: bus master, fast devsel, latency 0, IRQ 11, NUMA node 1
	Memory at fe840000 (32-bit, non-prefetchable) [size=4K]
	Memory at fc200000 (64-bit, prefetchable) [size=16K]
	Expansion ROM at fe800000 [disabled] [size=256K]
	Capabilities: [dc] MSI-X: Enable+ Count=3 Masked-
	Capabilities: [c8] Vendor Specific Information: VirtIO: <unknown>
	Capabilities: [b4] Vendor Specific Information: VirtIO: Notify
	Capabilities: [a4] Vendor Specific Information: VirtIO: DeviceCfg
	Capabilities: [94] Vendor Specific Information: VirtIO: ISR
	Capabilities: [84] Vendor Specific Information: VirtIO: CommonCfg
	Capabilities: [7c] Power Management version 3
	Capabilities: [40] Express Endpoint, MSI 00
	Kernel driver in use: virtio-pci
	Kernel modules: virtio_pci

68:03.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge (prog-if 00 [Normal decode])
	Flags: bus master, 66MHz, fast devsel, latency 0, IRQ 10, NUMA node 1
	Memory at fe200000 (64-bit, non-prefetchable) [size=256]
	Bus: primary=68, secondary=69, subordinate=69, sec-latency=0
	I/O behind bridge: 0000c000-0000cfff
	Memory behind bridge: fe000000-fe1fffff
	Prefetchable memory behind bridge: 00000000fc000000-00000000fc1fffff
	Capabilities: [4c] MSI: Enable- Count=1/1 Maskable+ 64bit+
	Capabilities: [48] Slot ID: 0 slots, First+, chassis 08
	Capabilities: [40] Hot-plug capable
	Kernel modules: shpchp

69:03.0 Ethernet controller: Red Hat, Inc Virtio network device
	Subsystem: Red Hat, Inc Device 0001
	Flags: bus master, fast devsel, latency 0, IRQ 10, NUMA node 1
	I/O ports at c000 [size=32]
	Memory at fe040000 (32-bit, non-prefetchable) [size=4K]
	Memory at fc000000 (64-bit, prefetchable) [size=16K]
	Expansion ROM at fe000000 [disabled] [size=256K]
	Capabilities: [98] MSI-X: Enable+ Count=3 Masked-
	Capabilities: [84] Vendor Specific Information: VirtIO: <unknown>
	Capabilities: [70] Vendor Specific Information: VirtIO: Notify
	Capabilities: [60] Vendor Specific Information: VirtIO: DeviceCfg
	Capabilities: [50] Vendor Specific Information: VirtIO: ISR
	Capabilities: [40] Vendor Specific Information: VirtIO: CommonCfg
	Kernel driver in use: virtio-pci
	Kernel modules: virtio_pci

2. negative test: attach pcie-root-port controller to pcie-expander-bus at slot 32
<controller type='pci' index='3' model='pcie-expander-bus'>
      <model name='pxb-pcie'/>
      <target busNr='100'>
        <node>1</node>
      </target>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='ioh3420'/>
      <target chassis='4' port='0x0'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x20' function='0x0'/>
    </controller>
error: XML error: Invalid PCI address slot='0x20', must be <= 0x1F
Failed. Try again? [y,n,i,f,?]: 

Both test scenarios get expected results. Tested under UEFI and passed as well.
Comment 8 errata-xmlrpc 2016-12-06 17:11:22 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2885.html