1442930 – [ppc64le] BARs are incorrectly assigned for some devices under P2P bridges

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1442930 - [ppc64le] BARs are incorrectly assigned for some devices under P2P bridges

Summary: [ppc64le] BARs are incorrectly assigned for some devices under P2P bridges

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	SLOF
Sub Component:
Version:	7.4
Hardware:	ppc64le
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	rc
Target Release:	---
Assignee:	Thomas Huth
QA Contact:	Qunfang Zhang
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2017-04-18 06:32 UTC by Qunfang Zhang
Modified:	2017-08-01 22:33 UTC (History)
CC List:	14 users (show)
Fixed In Version:	SLOF-20170303-3.git66d250e.el7
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-08-01 22:33:27 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2017:2093	0	normal	SHIPPED_LIVE	SLOF bug fix and enhancement update	2017-08-01 19:35:59 UTC

Description Qunfang Zhang 2017-04-18 06:32:47 UTC

Description of problem:
Boot a guest with usb mouse and usb keyboard while the usb controller is under pci-bridge, the mouse and keyboard can't work.  If usb controller is under pci bus, it works well.

Version-Release number of selected component (if applicable):
kernel-3.10.0-653.el7.ppc64le
qemu-kvm-rhev-2.9.0-0.el7.patchwork201703291116.ppc64le
SLOF-20170303-1.git66d250e.el7.noarch

How reproducible:
100%

Steps to Reproduce:
1. Boot guest with:

# /usr/libexec/qemu-kvm ... 
-device pci-bridge,bus=pci.0,id=bridge1,chassis_nr=1 -device nec-usb-xhci,id=xhci1,bus=bridge1,addr=0x3 -vnc :10 -vga std -device usb-kbd,bus=xhci1.0 -device usb-mouse,bus=xhci1.0

(Actually my full command line is:

# /usr/libexec/qemu-kvm -name test -machine pseries,accel=kvm,usb=off -m 4G -smp 4,sockets=1,cores=4,threads=1 -uuid 8aeab7e2-f341-4f8c-80e8-59e2968d85c2 -realtime mlock=off -nodefaults -monitor stdio -rtc base=utc -device pci-bridge,bus=pci.0,id=bridge1,chassis_nr=1 -drive file=rhel74-ppc64le-virtio-scsi.qcow2,if=none,id=drive_blk,format=qcow2,cache=none,werror=stop,rerror=stop  -device virtio-blk-pci,drive=drive_blk,id=device_blk,bus=bridge1,addr=0x1 -device virtio-scsi-pci,id=scsi1,bus=bridge1,addr=0x2  -drive file=disk/d1,if=none,id=drive_scsi,format=qcow2,cache=none,werror=stop,rerror=stop  -device scsi-hd,drive=drive_scsi,id=device_scsi,bus=scsi1.0 -device nec-usb-xhci,id=xhci1,bus=bridge1,addr=0x3  -drive file=disk/d2,if=none,id=drive_usb,format=qcow2,cache=none,werror=stop,rerror=stop  -device usb-storage,drive=drive_usb,id=device_usb,bus=xhci1.0 -netdev tap,id=tap10  -device virtio-net-pci,netdev=tap10,mac=9a:6a:6b:6c:6d:6e,bus=bridge1,addr=0x4 -vnc :10 -vga std -device usb-kbd,bus=xhci1.0 -device usb-mouse,bus=xhci1.0
)

2. Login guest and try to use mouse and keyboard inside guest
3.

Actual results:
Mouse and keyboard don't work

Expected results:
Mouse and keyboard should work

Additional info:

Comment 2 Qunfang Zhang 2017-04-18 06:33:40 UTC

This is a Power specific issue and can not reproduce on x86.

Comment 3 David Gibson 2017-04-19 05:49:39 UTC

Haven't had a chance to try reproducing this yet.  My first guess would be a problem in the interrupt routing across the bridge.

Comment 4 David Gibson 2017-04-20 06:07:54 UTC

Looks like BARs aren't being assigned, and so the XHCI isn't initializing at all.

This probably means no PCI devices will work under a P2P bridge.

Comment 5 David Gibson 2017-04-21 07:02:29 UTC

After further investigation it seems that the device's BAR *is* being set (by SLOF, I think).  But for some reason the guest isn't seeing it.

Further this does seem to be specific to xhci - a virtio-scsi-pci device shows up properly assigned BARs, even under a bridge.

Alexey (IBM) reports that the guest kernel is failing to ioremap() the XHCI's BAR.

Still investigating why.

This could well be a guest kernel bug, but I'm not sure yet.

Comment 6 David Gibson 2017-04-21 07:08:23 UTC

I discussed this online with BenH and Alexey Kardashevskiy.

It looks like BAR allocations are done inside SLOF (except for hotplug, where they're done by the guest).  We've talked about moving this to qemu, but haven't actually done so yet.

The problem seems to be that SLOF is allocating 64-bit BARs within a bridge's 64-bit window, which sounds right, but isn't quite.  In this case the XHCI has a non-prefetchable 64-bit BAR, but P2P bridges, the 64-bit window is always prefetchable.  It's not valid to put a non-prefetchable BAR within a prefetchable window, but it is valid to put a 64-bit BAR within a 32-bit window (at least on PCI, PCIe is more complicated).  Linux sees the prefetch attribute mismatch and fails the mapping, hence the subsequent errors.

We're looking at a SLOF patch for this in the short term.  Longer term we want to move BAR assignment to qemu where we will need similar logic.

Comment 7 Laurent Vivier 2017-04-25 10:49:17 UTC

I've tested the patch proposed by Alexey.

http://patchwork.ozlabs.org/patch/754084/

Before:

...
[    0.044857] PCI host bridge to bus 0000:00
[    0.044862] pci_bus 0000:00: root bus resource [io  0x10000-0x1ffff] (bus address [0x0000-0xffff])
[    0.044866] pci_bus 0000:00: root bus resource [mem 0x200080000000-0x2000ffffffff] (bus address [0x80000000-0xffffffff])
[    0.044870] pci_bus 0000:00: root bus resource [mem 0x210000000000-0x21ffffffffff]
[    0.044873] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.052085] IOMMU table initialized, virtual merging enabled
[    0.052100] iommu: Adding device 0000:00:00.0 to group 0
[    0.052123] iommu: Adding device 0000:00:01.0 to group 0
[    0.052163] iommu: Adding device 0000:00:02.0 to group 0
[    0.052602] iommu: Adding device 0000:01:03.0 to group 0
[    0.052989] PCI: Cannot allocate resource region 0 of device 0000:01:03.0, will remap
[    0.053028] PCI: Probing PCI hardware done
...
[    0.101273] pci 0000:01:03.0: can't enable device: BAR 0 [mem size 0x00004000] not assigned
[    0.101276] pci 0000:01:03.0: Can't enable PCI device, BIOS handoff failed.
...
[    0.504860] xhci_hcd 0000:01:03.0: can't enable device: BAR 0 [mem size 0x00004000] not assigned

After:

[    0.047231] PCI: Probing PCI hardware
[    0.047282] PCI host bridge to bus 0000:00
[    0.047310] pci_bus 0000:00: root bus resource [io  0x10000-0x1ffff] (bus address [0x0000-0xffff])
[    0.047367] pci_bus 0000:00: root bus resource [mem 0x200080000000-0x2000ffffffff] (bus address [0x80000000-0xffffffff])
[    0.047435] pci_bus 0000:00: root bus resource [mem 0x210000000000-0x21ffffffffff]
[    0.047484] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.054708] IOMMU table initialized, virtual merging enabled
[    0.054759] iommu: Adding device 0000:00:00.0 to group 0
[    0.054812] iommu: Adding device 0000:00:01.0 to group 0
[    0.054880] iommu: Adding device 0000:00:02.0 to group 0
[    0.055353] iommu: Adding device 0000:01:03.0 to group 0
[    0.055865] PCI: Probing PCI hardware done
...
[    0.508790] xhci_hcd 0000:01:03.0: xHCI Host Controller
[    0.508878] xhci_hcd 0000:01:03.0: new USB bus registered, assigned bus number 1
[    0.509243] xhci_hcd 0000:01:03.0: ibm,query-pe-dma-windows(2026) 11800 8000000 20000000 returned 0
[    0.509950] xhci_hcd 0000:01:03.0: ibm,create-pe-dma-window(2027) 11800 8000000 20000000 10 21 returned 0 (liobn = 0x80000001 starting addr = 8000000 0)
[    0.538269] xhci_hcd 0000:01:03.0: Using 64-bit direct DMA at offset 800000000000000
[    0.538673] xhci_hcd 0000:01:03.0: hcc params 0x00087001 hci version 0x100 quirks 0x00000014
[    0.539767] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[    0.539809] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    0.539858] usb usb1: Product: xHCI Host Controller
[    0.539891] usb usb1: Manufacturer: Linux 3.10.0-628.el7.ppc64le xhci-hcd
[    0.539932] usb usb1: SerialNumber: 0000:01:03.0
[    0.540062] hub 1-0:1.0: USB hub found
[    0.540123] hub 1-0:1.0: 4 ports detected
[    0.540319] xhci_hcd 0000:01:03.0: xHCI Host Controller
[    0.540392] xhci_hcd 0000:01:03.0: new USB bus registered, assigned bus number 2
[    0.540495] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
[    0.540561] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003
[    0.540602] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[    0.540650] usb usb2: Product: xHCI Host Controller
[    0.540683] usb usb2: Manufacturer: Linux 3.10.0-628.el7.ppc64le xhci-hcd
[    0.540724] usb usb2: SerialNumber: 0000:01:03.0
[    0.540831] hub 2-0:1.0: USB hub found
[    0.540890] hub 2-0:1.0: 4 ports detected
...
[    0.876246] usb 1-1: new high-speed USB device number 2 using xhci_hcd
[    1.016636] usb 1-1: New USB device found, idVendor=0627, idProduct=0001
[    1.016687] usb 1-1: New USB device strings: Mfr=1, Product=4, SerialNumber=5
[    1.016736] usb 1-1: Product: QEMU USB Keyboard
[    1.016770] usb 1-1: Manufacturer: QEMU
[    1.016795] usb 1-1: SerialNumber: 42
[    1.017576] input: QEMU QEMU USB Keyboard as /devices/pci0000:00/0000:00:02.0/0000:01:03.0/usb1/1-1/1-1:1.0/input/input0
[    1.076369] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:01:03.0-1/input0
[    1.196229] usb 1-2: new high-speed USB device number 3 using xhci_hcd
[    1.336615] usb 1-2: New USB device found, idVendor=0627, idProduct=0001
[    1.336660] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=5
[    1.336709] usb 1-2: Product: QEMU USB Mouse
[    1.336742] usb 1-2: Manufacturer: QEMU
[    1.336768] usb 1-2: SerialNumber: 42
[    1.337368] input: QEMU QEMU USB Mouse as /devices/pci0000:00/0000:00:02.0/0000:01:03.0/usb1/1-2/1-2:1.0/input/input1
[    1.337599] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Mouse] on usb-0000:01:03.0-2/input0
...

Comment 8 Miroslav Rezanina 2017-05-05 08:56:29 UTC

Fix included in SLOF-20170303-3.git66d250e.el7

Comment 10 Qunfang Zhang 2017-05-09 08:45:28 UTC

Verified this bug with SLOF-20170303-3.git66d250e.el7.noarch.rpm with the same steps as comment 0, the issue is fixed.

Comment 11 errata-xmlrpc 2017-08-01 22:33:27 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:2093

Note You need to log in before you can comment on or make changes to this bug.