Bug 796677
Summary: | [XEN] Xen PCI-passthrough does not work with a Emulex Saturn-X LightPulse | ||||||
---|---|---|---|---|---|---|---|
Product: | Red Hat Enterprise Linux 5 | Reporter: | asilva <asilva> | ||||
Component: | xen | Assignee: | Xen Maintainance List <xen-maint> | ||||
Status: | CLOSED DUPLICATE | QA Contact: | Virtualization Bugs <virt-bugs> | ||||
Severity: | medium | Docs Contact: | |||||
Priority: | medium | ||||||
Version: | 5.7 | CC: | drjones, jmunilla, lersek, revers, xen-maint | ||||
Target Milestone: | rc | ||||||
Target Release: | --- | ||||||
Hardware: | All | ||||||
OS: | All | ||||||
Whiteboard: | |||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | Environment: | ||||||
Last Closed: | 2012-02-27 18:58:49 UTC | Type: | --- | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Attachments: |
|
Description
asilva
2012-02-23 13:01:48 UTC
The error message "ioremap failed for SLIM memory" is printed by lpfc_sli_pci_mem_setup() [drivers/scsi/lpfc/lpfc_init.c, 2.6.18-274.17.1.el5]: 5293 /* Get the bus address of Bar0 and Bar2 and the number of bytes 5294 * required by each mapping. 5295 */ 5296 phba->pci_bar0_map = pci_resource_start(pdev, 0); 5297 bar0map_len = pci_resource_len(pdev, 0); 5298 5299 phba->pci_bar2_map = pci_resource_start(pdev, 2); 5300 bar2map_len = pci_resource_len(pdev, 2); 5301 5302 /* Map HBA SLIM to a kernel virtual address. */ 5303 phba->slim_memmap_p = ioremap(phba->pci_bar0_map, bar0map_len); 5304 if (!phba->slim_memmap_p) { 5305 dev_printk(KERN_ERR, &pdev->dev, 5306 "ioremap failed for SLIM memory.\n"); 5307 goto out; 5308 } According to the Sep 23, 2011 domU config file attached to the CP case, the guest is PV. (This seems consistent with the tendency that HVM guests get passed-through devices as 06:00.0 and 07:00.0, IIRC, but the BDFs reported in comment 0 are different.) First, PCI passthrough to a PV guest is insecure (guest could setup a DMA wherever it wants). Second, the guest kernel is thus a xenified kernel, which diverts ioremap() as follows: ioremap() [include/asm-x86_64/mach-xen/asm/io.h] -> __ioremap() [arch/i386/mm/ioremap-xen.c] -> get_vm_area() [mm/vmalloc.c] -> __direct_remap_pfn_range() [arch/i386/mm/ioremap-xen.c] -> HYPERVISOR_mmu_update() After some checks, __ioremap() grabs a virtual address range, then __direct_remap_pfn_range() kicks the hypervisor in a batched loop to point the "init_mm" PTEs, covering the vaddr range, to the requested machine frames. Either the early checks fire, or the hypervisor refuses one of the PTE updates. "lspci -v -v -v" on the reproducer machine mentioned in comment 2 reports the following regions: 1a:00.0 Fibre Channel: Emulex Corporation Saturn-X: LightPulse Fibre Channel Host Adapter (rev 03) Region 0: Memory at 97a08000 (64-bit, non-prefetchable) [size=4K] Region 2: Memory at 97a00000 (64-bit, non-prefetchable) [size=16K] Region 4: I/O ports at 2100 [size=256] 1a:00.1 Fibre Channel: Emulex Corporation Saturn-X: LightPulse Fibre Channel Host Adapter (rev 03) Region 0: Memory at 97a09000 (64-bit, non-prefetchable) [size=4K] Region 2: Memory at 97a04000 (64-bit, non-prefetchable) [size=16K] Region 4: I/O ports at 2000 [size=256] According to the error message and the lpfc_sli_pci_mem_setup() source, region 0 is "SLIM memory", while region 2 is "HBA control registers". So the xenified ioremap() fails to map 97a08000..+4K and 97a09000..+4K. I would recommend: - checking 97a08000 against the Xen E820 map (xm dmesg) -- does the card specify a region that Xen considers reserved RAM (or RAM at all)? - testing the PV guest with RHEL-5.8 GA in both host and guest - passthrough to an HVM guest (with iommu enabled) instead of the PV guest - if none of those work, I'll add debug logging to __ioremap() (see above) and figure out where exactly it fails. Created attachment 565629 [details] Xen __ioremap() debugging (In reply to comment #3) > I would recommend: > > - checking 97a08000 against the Xen E820 map (xm dmesg) -- does the card > specify a region that Xen considers reserved RAM (or RAM at all)? > > - testing the PV guest with RHEL-5.8 GA in both host and guest > > - passthrough to an HVM guest (with iommu enabled) instead of the PV guest > > - if none of those work, I'll add debug logging to __ioremap() (see above) > and figure out where exactly it fails. For the fourth option. Reproduced the bug in-house with the x86_64 -308 xen kernel (both domU and dom0). When the guest (domid == 1) logs (during boot): PCI: Enabling device 0000:00:00.0 (0000 -> 0002) PCI: Setting latency timer of device 0000:00:00.0 to 64 lpfc 0000:00:00.0: ioremap failed for SLIM memory. lpfc 0000:00:00.0: 0:1402 Failed to set up pci memory space. The hypervisor prints: (XEN) mm.c:630:d1 Non-privileged (1) attempt to map I/O space 00097a08 Referring back to comment 3: > Region 0: Memory at 97a08000 (64-bit, non-prefetchable) [size=4K] > [...] > According to the error message and the lpfc_sli_pci_mem_setup() source, > region 0 is "SLIM memory", Machine frame 97a08 corresponds to this mach addr. Adding vendor-id:device-id ('10df:f100') to /etc/xen/xend-pci-permissive.sxp only takes care of the PCI conf space access: pciback 0000:1a:00.0: enabling permissive mode configuration space accesses! pciback 0000:1a:00.0: permissive mode is potentially unsafe! pciback 0000:1a:00.1: enabling permissive mode configuration space accesses! pciback 0000:1a:00.1: permissive mode is potentially unsafe! but the hypervisor keeps complaining about the non-privileged mapping attempt. This bug is a duplicate of bug 735890. See in particular: - bug 735890 comment 15, - bug 735890 comment 16, - bug 735890 comment 17. Applying that analysis to the current values: [root@in-house-reproducer ~]# cat -n \ /sys/bus/pci/devices/0000:1a:00.0/resource 1 0x0000000097a08000 0x0000000097a08fff 0x0000000000020204 2 0x0000000000000000 0x0000000000000000 0x0000000000000000 3 0x0000000097a00000 0x0000000097a03fff 0x0000000000020204 4 0x0000000000000000 0x0000000000000000 0x0000000000000000 5 0x0000000000002100 0x00000000000021ff 0x0000000000020101 6 0x0000000000000000 0x0000000000000000 0x0000000000000000 7 0x0000000098200000 0x000000009823ffff 0x0000000000027200 From those, line 1, 3, 7 are iomem ranges, while line 5 describes an ioport range. Other lines are ignored. The following is a xend log excerpt, written at domU startup: Unconstrained device: 0000:1a:00.0 pci: enabling ioport 0x2100/0x100 pci: enabling iomem 0x97a08000/0x1000 pfn 0x97a08/0x1 pci: enabling iomem 0x97a00000/0x4000 pfn 0x97a00/0x4 pci: enabling iomem 0x98200000/0x40000 pfn 0x98200/0x40 pci-msix: remove permission for 0x97a02000/0x20000 0x97a02/0x20 pci-msix: remove permission for 0x97a03000/0x1000 0x97a03/0x1 Note that "iomem 0x97a08000/0x1000" from the above corresponds to Region 0: Memory at 97a08000 (64-bit, non-prefetchable) [size=4K] from comment 3, and to (XEN) mm.c:630:d1 Non-privileged (1) attempt to map I/O space 00097a08 from comment 7. When xend removes the permission for the MSI-X range 0x97a02000/0x20000 (see duplicate bug 735890), it kills any previously granted permission for included/overlapping iomem regions: 0x97a02000 -- start of MSI-X iomem (inclusive) 0x97a08000 -- start of lpfc SLI memory (inclusive) 0x97a09000 -- end of lpfc SLI memory (exclusive), size 0x1000 0x97a22000 -- end of MSI-X iomem (exclusive), size 0x20000 which causes lpfc_sli_pci_mem_setup() to fail in the guest. *** This bug has been marked as a duplicate of bug 735890 *** |