Bug 1442930
| Summary: | [ppc64le] BARs are incorrectly assigned for some devices under P2P bridges | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Qunfang Zhang <qzhang> |
| Component: | SLOF | Assignee: | Thomas Huth <thuth> |
| Status: | CLOSED ERRATA | QA Contact: | Qunfang Zhang <qzhang> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 7.4 | CC: | dgibson, knoel, lvivier, mdeng, michen, mrezanin, thuth, virt-maint, xianwang, xuma, yduan, yhong, yilzhang, zhengtli |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | ppc64le | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | SLOF-20170303-3.git66d250e.el7 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2017-08-01 22:33:27 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Qunfang Zhang
2017-04-18 06:32:47 UTC
This is a Power specific issue and can not reproduce on x86. Haven't had a chance to try reproducing this yet. My first guess would be a problem in the interrupt routing across the bridge. Looks like BARs aren't being assigned, and so the XHCI isn't initializing at all. This probably means no PCI devices will work under a P2P bridge. After further investigation it seems that the device's BAR *is* being set (by SLOF, I think). But for some reason the guest isn't seeing it. Further this does seem to be specific to xhci - a virtio-scsi-pci device shows up properly assigned BARs, even under a bridge. Alexey (IBM) reports that the guest kernel is failing to ioremap() the XHCI's BAR. Still investigating why. This could well be a guest kernel bug, but I'm not sure yet. I discussed this online with BenH and Alexey Kardashevskiy. It looks like BAR allocations are done inside SLOF (except for hotplug, where they're done by the guest). We've talked about moving this to qemu, but haven't actually done so yet. The problem seems to be that SLOF is allocating 64-bit BARs within a bridge's 64-bit window, which sounds right, but isn't quite. In this case the XHCI has a non-prefetchable 64-bit BAR, but P2P bridges, the 64-bit window is always prefetchable. It's not valid to put a non-prefetchable BAR within a prefetchable window, but it is valid to put a 64-bit BAR within a 32-bit window (at least on PCI, PCIe is more complicated). Linux sees the prefetch attribute mismatch and fails the mapping, hence the subsequent errors. We're looking at a SLOF patch for this in the short term. Longer term we want to move BAR assignment to qemu where we will need similar logic. I've tested the patch proposed by Alexey. http://patchwork.ozlabs.org/patch/754084/ Before: ... [ 0.044857] PCI host bridge to bus 0000:00 [ 0.044862] pci_bus 0000:00: root bus resource [io 0x10000-0x1ffff] (bus address [0x0000-0xffff]) [ 0.044866] pci_bus 0000:00: root bus resource [mem 0x200080000000-0x2000ffffffff] (bus address [0x80000000-0xffffffff]) [ 0.044870] pci_bus 0000:00: root bus resource [mem 0x210000000000-0x21ffffffffff] [ 0.044873] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.052085] IOMMU table initialized, virtual merging enabled [ 0.052100] iommu: Adding device 0000:00:00.0 to group 0 [ 0.052123] iommu: Adding device 0000:00:01.0 to group 0 [ 0.052163] iommu: Adding device 0000:00:02.0 to group 0 [ 0.052602] iommu: Adding device 0000:01:03.0 to group 0 [ 0.052989] PCI: Cannot allocate resource region 0 of device 0000:01:03.0, will remap [ 0.053028] PCI: Probing PCI hardware done ... [ 0.101273] pci 0000:01:03.0: can't enable device: BAR 0 [mem size 0x00004000] not assigned [ 0.101276] pci 0000:01:03.0: Can't enable PCI device, BIOS handoff failed. ... [ 0.504860] xhci_hcd 0000:01:03.0: can't enable device: BAR 0 [mem size 0x00004000] not assigned After: [ 0.047231] PCI: Probing PCI hardware [ 0.047282] PCI host bridge to bus 0000:00 [ 0.047310] pci_bus 0000:00: root bus resource [io 0x10000-0x1ffff] (bus address [0x0000-0xffff]) [ 0.047367] pci_bus 0000:00: root bus resource [mem 0x200080000000-0x2000ffffffff] (bus address [0x80000000-0xffffffff]) [ 0.047435] pci_bus 0000:00: root bus resource [mem 0x210000000000-0x21ffffffffff] [ 0.047484] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.054708] IOMMU table initialized, virtual merging enabled [ 0.054759] iommu: Adding device 0000:00:00.0 to group 0 [ 0.054812] iommu: Adding device 0000:00:01.0 to group 0 [ 0.054880] iommu: Adding device 0000:00:02.0 to group 0 [ 0.055353] iommu: Adding device 0000:01:03.0 to group 0 [ 0.055865] PCI: Probing PCI hardware done ... [ 0.508790] xhci_hcd 0000:01:03.0: xHCI Host Controller [ 0.508878] xhci_hcd 0000:01:03.0: new USB bus registered, assigned bus number 1 [ 0.509243] xhci_hcd 0000:01:03.0: ibm,query-pe-dma-windows(2026) 11800 8000000 20000000 returned 0 [ 0.509950] xhci_hcd 0000:01:03.0: ibm,create-pe-dma-window(2027) 11800 8000000 20000000 10 21 returned 0 (liobn = 0x80000001 starting addr = 8000000 0) [ 0.538269] xhci_hcd 0000:01:03.0: Using 64-bit direct DMA at offset 800000000000000 [ 0.538673] xhci_hcd 0000:01:03.0: hcc params 0x00087001 hci version 0x100 quirks 0x00000014 [ 0.539767] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002 [ 0.539809] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.539858] usb usb1: Product: xHCI Host Controller [ 0.539891] usb usb1: Manufacturer: Linux 3.10.0-628.el7.ppc64le xhci-hcd [ 0.539932] usb usb1: SerialNumber: 0000:01:03.0 [ 0.540062] hub 1-0:1.0: USB hub found [ 0.540123] hub 1-0:1.0: 4 ports detected [ 0.540319] xhci_hcd 0000:01:03.0: xHCI Host Controller [ 0.540392] xhci_hcd 0000:01:03.0: new USB bus registered, assigned bus number 2 [ 0.540495] usb usb2: We don't know the algorithms for LPM for this host, disabling LPM. [ 0.540561] usb usb2: New USB device found, idVendor=1d6b, idProduct=0003 [ 0.540602] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1 [ 0.540650] usb usb2: Product: xHCI Host Controller [ 0.540683] usb usb2: Manufacturer: Linux 3.10.0-628.el7.ppc64le xhci-hcd [ 0.540724] usb usb2: SerialNumber: 0000:01:03.0 [ 0.540831] hub 2-0:1.0: USB hub found [ 0.540890] hub 2-0:1.0: 4 ports detected ... [ 0.876246] usb 1-1: new high-speed USB device number 2 using xhci_hcd [ 1.016636] usb 1-1: New USB device found, idVendor=0627, idProduct=0001 [ 1.016687] usb 1-1: New USB device strings: Mfr=1, Product=4, SerialNumber=5 [ 1.016736] usb 1-1: Product: QEMU USB Keyboard [ 1.016770] usb 1-1: Manufacturer: QEMU [ 1.016795] usb 1-1: SerialNumber: 42 [ 1.017576] input: QEMU QEMU USB Keyboard as /devices/pci0000:00/0000:00:02.0/0000:01:03.0/usb1/1-1/1-1:1.0/input/input0 [ 1.076369] hid-generic 0003:0627:0001.0001: input,hidraw0: USB HID v1.11 Keyboard [QEMU QEMU USB Keyboard] on usb-0000:01:03.0-1/input0 [ 1.196229] usb 1-2: new high-speed USB device number 3 using xhci_hcd [ 1.336615] usb 1-2: New USB device found, idVendor=0627, idProduct=0001 [ 1.336660] usb 1-2: New USB device strings: Mfr=1, Product=2, SerialNumber=5 [ 1.336709] usb 1-2: Product: QEMU USB Mouse [ 1.336742] usb 1-2: Manufacturer: QEMU [ 1.336768] usb 1-2: SerialNumber: 42 [ 1.337368] input: QEMU QEMU USB Mouse as /devices/pci0000:00/0000:00:02.0/0000:01:03.0/usb1/1-2/1-2:1.0/input/input1 [ 1.337599] hid-generic 0003:0627:0001.0002: input,hidraw1: USB HID v0.01 Mouse [QEMU QEMU USB Mouse] on usb-0000:01:03.0-2/input0 ... Fix included in SLOF-20170303-3.git66d250e.el7 Verified this bug with SLOF-20170303-3.git66d250e.el7.noarch.rpm with the same steps as comment 0, the issue is fixed. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2093 |