Bug 1443433
| Summary: | [ppc64le] Guest failed to boot up with 8 nested pci-bridge | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Qunfang Zhang <qzhang> | ||||
| Component: | SLOF | Assignee: | Thomas Huth <thuth> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Qunfang Zhang <qzhang> | ||||
| Severity: | low | Docs Contact: | |||||
| Priority: | low | ||||||
| Version: | 7.4 | CC: | bugproxy, dgibson, fnovak, hannsj_uhl, knoel, lvivier, mdeng, michen, mrezanin, mtessun, qzhang, thuth, virt-maint, xianwang, xuma, yduan, yhong, yilzhang, zhengtli | ||||
| Target Milestone: | rc | ||||||
| Target Release: | 7.4 | ||||||
| Hardware: | ppc64le | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | SLOF-20170303-4.git66d250e.el7 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2017-08-01 22:33:27 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1299988 | ||||||
| Attachments: |
|
||||||
|
Description
Qunfang Zhang
2017-04-19 09:15:44 UTC
This is a Power specific issue and can't reproduce on x86 (1) If guest OS image is under 1 of the pci-bridge, I can hit this issue. (2) If guest OS image is attached under pci.0 bus, and attach 8 additional data disk under 8 nested pci-bridge, guest can boot up successfully. So it works well for the following command line: # /usr/libexec/qemu-kvm -name test -machine pseries,accel=kvm,usb=off -m 8G -smp 8,sockets=2,cores=4,threads=1 -uuid 8aeab7e2-f341-4f8c-80e8-59e2968d85c2 -realtime mlock=off -nodefaults -monitor stdio -rtc base=utc -drive file=rhel74-ppc64le-virtio-scsi.qcow2,if=none,id=drive1,format=qcow2,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive1,id=virtio1,bus=pci.0 -device pci-bridge,id=bridge1,chassis_nr=1,bus=pci.0 -drive file=/home/images/storage1.qcow2,if=none,id=drive-virtio-disk1,format=qcow2,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio-disk1,id=virtio-disk1,bus=bridge1,addr=0x1 -device pci-bridge,id=bridge2,chassis_nr=2,bus=bridge1,addr=0x2 -drive file=/home/images/storage2.qcow2,if=none,id=drive-virtio-disk2,format=qcow2,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio-disk2,id=virtio-disk2,bus=bridge2,addr=0x1 -device nec-usb-xhci,id=xhci1,bus=pci.0,addr=0xa -vnc :10 -vga std -chardev socket,id=console0,path=/tmp/console0,server,nowait -device spapr-vty,chardev=console0 -device usb-kbd -device usb-tablet -device pci-bridge,id=bridge3,chassis_nr=3,bus=bridge2,addr=0x2 -drive file=/home/images/storage3.qcow2,if=none,id=drive-virtio-disk3,format=qcow2,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio-disk3,id=virtio-disk3,bus=bridge3,addr=0x1 -device pci-bridge,id=bridge4,chassis_nr=4,bus=bridge3,addr=0x2 -drive file=/home/images/storage4.qcow2,if=none,id=drive-virtio-disk4,format=qcow2,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio-disk4,id=virtio-disk4,bus=bridge4,addr=0x1 -device pci-bridge,id=bridge5,chassis_nr=5,bus=bridge4,addr=0x2 -drive file=/home/images/storage5.qcow2,if=none,id=drive-virtio-disk5,format=qcow2,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio-disk5,id=virtio-disk5,bus=bridge5,addr=0x1 -device pci-bridge,id=bridge6,chassis_nr=6,bus=bridge5,addr=0x2 -drive file=/home/images/storage6.qcow2,if=none,id=drive-virtio-disk6,format=qcow2,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio-disk6,id=virtio-disk6,bus=bridge6,addr=0x1 -device pci-bridge,id=bridge7,chassis_nr=7,bus=bridge6,addr=0x2 -drive file=/home/images/storage7.qcow2,if=none,id=drive-virtio-disk7,format=qcow2,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio-disk7,id=virtio-disk7,bus=bridge7,addr=0x1 -device pci-bridge,id=bridge8,chassis_nr=8,bus=bridge7,addr=0x2 -drive file=/home/images/storage8.qcow2,if=none,id=drive-virtio-disk8,format=qcow2,cache=none,werror=stop,rerror=stop -device virtio-blk-pci,drive=drive-virtio-disk8,id=virtio-disk8,bus=bridge8,addr=0x1 Created attachment 1272530 [details]
Screenshot of guest
Probably not much point trying to debug this until we've fixed 1442930. As it stands almost nothing is going to work behind a P2P bridge. Some details, the guest kernel logs: [ 0.051433] PCI: Probing PCI hardware [ 0.051485] PCI host bridge to bus 0000:00 [ 0.051513] pci_bus 0000:00: root bus resource [io 0x10000-0x1ffff] (bus address [0x0000-0xffff]) [ 0.051570] pci_bus 0000:00: root bus resource [mem 0x200080000000-0x2000ffffffff] (bus address [0x80000000-0xffffffff]) [ 0.051634] pci_bus 0000:00: root bus resource [mem 0x210000000000-0x21ffffffffff] [ 0.051683] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.058243] IOMMU table initialized, virtual merging enabled [ 0.058294] iommu: Adding device 0000:00:00.0 to group 0 [ 0.059260] iommu: Adding device 0000:01:01.0 to group 0 [ 0.059331] iommu: Adding device 0000:01:02.0 to group 0 [ 0.060284] iommu: Adding device 0000:02:01.0 to group 0 [ 0.060351] iommu: Adding device 0000:02:02.0 to group 0 [ 0.061313] iommu: Adding device 0000:03:01.0 to group 0 [ 0.061376] iommu: Adding device 0000:03:02.0 to group 0 [ 0.062349] iommu: Adding device 0000:04:01.0 to group 0 [ 0.062415] iommu: Adding device 0000:04:02.0 to group 0 [ 0.063382] iommu: Adding device 0000:05:01.0 to group 0 [ 0.063450] iommu: Adding device 0000:05:02.0 to group 0 [ 0.064435] iommu: Adding device 0000:06:01.0 to group 0 [ 0.064508] iommu: Adding device 0000:06:02.0 to group 0 [ 0.065482] iommu: Adding device 0000:07:01.0 to group 0 [ 0.065557] iommu: Adding device 0000:07:02.0 to group 0 [ 0.066287] iommu: Adding device 0000:08:01.0 to group 0 [ 0.066524] PCI: Cannot allocate resource region 0 of device 0000:01:01.0, will remap [ 0.066589] PCI: Cannot allocate resource region 0 of device 0000:02:01.0, will remap [ 0.066654] PCI: Cannot allocate resource region 0 of device 0000:03:01.0, will remap [ 0.066720] PCI: Cannot allocate resource region 0 of device 0000:04:01.0, will remap [ 0.066785] PCI: Cannot allocate resource region 0 of device 0000:05:01.0, will remap [ 0.066850] PCI: Cannot allocate resource region 0 of device 0000:06:01.0, will remap [ 0.066915] PCI: Cannot allocate resource region 0 of device 0000:07:01.0, will remap [ 0.066981] PCI: Cannot allocate resource region 0 of device 0000:08:01.0, will remap ... [ 0.733664] virtio-pci 0000:01:01.0: can't enable device: BAR 0 [io size 0x0040] not assigned [ 0.733737] virtio-pci: probe of 0000:01:01.0 failed with error -22 [ 0.733823] virtio-pci 0000:02:01.0: can't enable device: BAR 0 [io size 0x0040] not assigned [ 0.733882] virtio-pci: probe of 0000:02:01.0 failed with error -22 [ 0.733964] virtio-pci 0000:03:01.0: can't enable device: BAR 0 [io size 0x0040] not assigned [ 0.734027] virtio-pci: probe of 0000:03:01.0 failed with error -22 [ 0.734116] virtio-pci 0000:04:01.0: can't enable device: BAR 0 [io size 0x0040] not assigned [ 0.734176] virtio-pci: probe of 0000:04:01.0 failed with error -22 [ 0.734274] virtio-pci 0000:05:01.0: can't enable device: BAR 0 [io size 0x0040] not assigned [ 0.734356] virtio-pci: probe of 0000:05:01.0 failed with error -22 [ 0.734442] virtio-pci 0000:06:01.0: can't enable device: BAR 0 [io size 0x0040] not assigned [ 0.734504] virtio-pci: probe of 0000:06:01.0 failed with error -22 [ 0.734586] virtio-pci 0000:07:01.0: can't enable device: BAR 0 [io size 0x0040] not assigned [ 0.734645] virtio-pci: probe of 0000:07:01.0 failed with error -22 [ 0.734730] virtio-pci 0000:08:01.0: can't enable device: BAR 0 [io size 0x0040] not assigned [ 0.734788] virtio-pci: probe of 0000:08:01.0 failed with error -22 ... The fix proposed for BZ1442930 doesn't work for this one. The difference is for XHCI this is mem BAR, whereas here this is io BAR. Using "disable-legacy=on" with virtio-blk-pci devices seems to workaround the problem. Could you try to reproduce the problem on x86_64? It seems the problem is caused by an x86 legacy limitation (max of 65536 IO ports). (In reply to Laurent Vivier from comment #8) > Could you try to reproduce the problem on x86_64? > > It seems the problem is caused by an x86 legacy limitation (max of 65536 IO > ports). According to comment 2, this bug could not be reproduced on x86. :) I have compared the I/O mapping between ppc64le (with 7 pci-bridges) and x86_64 (with 8): ppc64le: # lspci -v|grep "I/O behind bridge: " I/O behind bridge: 00001000-0000efff I/O behind bridge: 00003000-0000efff I/O behind bridge: 00005000-0000efff I/O behind bridge: 00007000-0000efff I/O behind bridge: 00009000-0000efff I/O behind bridge: 0000b000-0000efff I/O behind bridge: 0000d000-0000efff So it seems the base address for the 8th pci-bridge is beyond 0xefff, that explains why it doesn't work. x86_64: # lspci -v|grep "I/O behind bridge: " I/O behind bridge: 00001000-00008fff I/O behind bridge: 00001000-00007fff I/O behind bridge: 00001000-00006fff I/O behind bridge: 00001000-00005fff I/O behind bridge: 00001000-00004fff I/O behind bridge: 00001000-00003fff I/O behind bridge: 00001000-00002fff I/O behind bridge: 00001000-00001fff It seems x86_64 uses as twice less of I/O space/bridge. So it should be able to manage 16 pci-bridges. But I've tested with 9 pci-bridges and the BIOS hangs... x86 is using less space per bridge, but the range 0x9000..0xefff is already missing from the outermost P2P bridge's range, so it's got substantially less space to work with. 0x1000..0x8fff / 0x1000 per-bridge gives you exactly 8 bridges which matches the results you've seen. My first guess would be that 0x9000..0xefff is being allocated directly to integrated legacy devices on the root bus. Sounds like we might want to change SLOF's allocation scheme to only give 4kiB per bridge instead of 8kiB as it appears to now. If 4kiB is enough on x86, it ought to be enough on Power as well. (In reply to David Gibson from comment #11) ... > Sounds like we might want to change SLOF's allocation scheme to only give > 4kiB per bridge instead of 8kiB as it appears to now. If 4kiB is enough on > x86, it ought to be enough on Power as well. pci-bridges have already 4KiB of I/O, except they must be aligned to a 4KiB address. If you don't put any device on the bridge, you can add 15 pci-bridges: # lspci -v |grep "I/O behind bridge: " I/O behind bridge: 00001000-0000ffff I/O behind bridge: 00002000-0000ffff I/O behind bridge: 00003000-0000ffff I/O behind bridge: 00004000-0000ffff I/O behind bridge: 00005000-0000ffff I/O behind bridge: 00006000-0000ffff I/O behind bridge: 00007000-0000ffff I/O behind bridge: 00008000-0000ffff I/O behind bridge: 00009000-0000ffff I/O behind bridge: 0000a000-0000ffff I/O behind bridge: 0000b000-0000ffff I/O behind bridge: 0000c000-0000ffff I/O behind bridge: 0000d000-0000ffff I/O behind bridge: 0000e000-0000ffff I/O behind bridge: 0000f000-0000ffff But if you add a virtio-pci device on the pci-bridge, this devices needs a 64B I/O, but as the following I/O space is a bridge, this space must be aligned to a 4KiB address (by SLOF). -- pci-bridge | +--------------- 0x0000 | virtio-lblk +--------------- 0x0040 | | empty | +--------------- 0x1000 | pci-bridge | | | +------------------- 0x0000 | | virtio-blk | +------------------- 0x0040 | | | | empty | | | +------------------- 0x1000 | | | | pci-bridge | | | ......... | +------------------- 0x2000 | +--------------- 0x3000 So the pci-bridge needs a 8 KiB I/O, because there is inside another pci-bridge that needs to be aligned to a 4 KiB and another devices. We can also see this with QEMU HMP command "info mtree" (simplified): - pci-bridges without PCI devices: memory-region: pci_bridge_io 0000000000000000-000000000000ffff (prio 0, i/o): pci_bridge_io 0000000000002000-0000000000007fff (prio 1, i/o) memory-region: pci_bridge_io 0000000000000000-000000000000ffff (prio 0, i/o): pci_bridge_io 0000000000003000-0000000000007fff (prio 1, i/o) memory-region: pci_bridge_io 0000000000000000-000000000000ffff (prio 0, i/o): pci_bridge_io 0000000000004000-0000000000007fff (prio 1, i/o) memory-region: pci_bridge_io 0000000000000000-000000000000ffff (prio 0, i/o): pci_bridge_io 0000000000005000-0000000000007fff (prio 1, i/o) memory-region: pci_bridge_io 0000000000000000-000000000000ffff (prio 0, i/o): pci_bridge_io 0000000000006000-0000000000007fff (prio 1, i/o) memory-region: pci_bridge_io 0000000000000000-000000000000ffff (prio 0, i/o): pci_bridge_io 0000000000007000-0000000000007fff (prio 1, i/o) memory-region: pci_bridge_io 0000000000000000-000000000000ffff (prio 0, i/o): pci_bridge_io - with virtio-device inside pci-bridges: memory-region: pci_bridge_io 0000000000000000-000000000000ffff (prio 0, i/o): pci_bridge_io 0000000000002000-000000000000203f (prio 1, i/o): virtio-pci 0000000000003000-000000000000efff (prio 1, i/o) memory-region: pci_bridge_io 0000000000000000-000000000000ffff (prio 0, i/o): pci_bridge_io 0000000000004000-000000000000403f (prio 1, i/o): virtio-pci 0000000000005000-000000000000efff (prio 1, i/o) memory-region: pci_bridge_io 0000000000000000-000000000000ffff (prio 0, i/o): pci_bridge_io 0000000000006000-000000000000603f (prio 1, i/o): virtio-pci 0000000000007000-000000000000efff (prio 1, i/o) memory-region: pci_bridge_io 0000000000000000-000000000000ffff (prio 0, i/o): pci_bridge_io 0000000000008000-000000000000803f (prio 1, i/o): virtio-pci 0000000000009000-000000000000efff (prio 1, i/o) memory-region: pci_bridge_io 0000000000000000-000000000000ffff (prio 0, i/o): pci_bridge_io 000000000000a000-000000000000a03f (prio 1, i/o): virtio-pci 000000000000b000-000000000000efff (prio 1, i/o) memory-region: pci_bridge_io 0000000000000000-000000000000ffff (prio 0, i/o): pci_bridge_io 000000000000c000-000000000000c03f (prio 1, i/o): virtio-pci 000000000000d000-000000000000efff (prio 1, i/o) memory-region: pci_bridge_io 0000000000000000-000000000000ffff (prio 0, i/o): pci_bridge_io 000000000000e000-000000000000e03f (prio 1, i/o): virtio-pci PC BIOS seems to allocate I/O addresses recursively while SLOF seems to do it iteratively. This avoids to waste memory for alignment on each bridge.
memory-region: pci_bridge_io
0000000000000000-000000000000ffff (prio 0, i/o): pci_bridge_io
0000000000001000-0000000000005fff (prio 1, i/o)
0000000000006000-000000000000603f (prio 1, i/o): virtio-pci
memory-region: pci_bridge_io
0000000000000000-000000000000ffff (prio 0, i/o): pci_bridge_io
0000000000001000-0000000000004fff (prio 1, i/o)
0000000000005000-000000000000503f (prio 1, i/o): virtio-pci
memory-region: pci_bridge_io
0000000000000000-000000000000ffff (prio 0, i/o): pci_bridge_io
0000000000001000-0000000000003fff (prio 1, i/o)
0000000000004000-000000000000403f (prio 1, i/o): virtio-pci
memory-region: pci_bridge_io
0000000000000000-000000000000ffff (prio 0, i/o): pci_bridge_io
0000000000001000-0000000000002fff (prio 1, i/o)
0000000000003000-000000000000303f (prio 1, i/o): virtio-pci
memory-region: pci_bridge_io
0000000000000000-000000000000ffff (prio 0, i/o): pci_bridge_io
0000000000001000-0000000000001fff (prio 1, i/o)
0000000000002000-000000000000203f (prio 1, i/o): virtio-pci
memory-region: pci_bridge_io
0000000000000000-000000000000ffff (prio 0, i/o): pci_bridge_io
0000000000001000-000000000000103f (prio 1, i/o): virtio-pci
As we can't reduce the size of the I/O space because of the alignment requirement of pci-bridges, I add a dependency on BZ#1344299 as suggested by Laszlo Ersek <lersek>. I'm not really understanding the relevance of bug 1344299, since we're dealing with a PCI, not PCIe heirarchy here. Nor do I really understand the distinction you're making between "recursive" and "iterative" allocation. Looking at the dumps above, is it just a question that SLOF is allocating the virtio device then the bridge at each level, whereas PC bios is allocating the bridge then the virtio device? (In reply to David Gibson from comment #15) > I'm not really understanding the relevance of bug 1344299, since we're > dealing with a PCI, not PCIe heirarchy here. Yes, you're right. Removing. > Nor do I really understand the distinction you're making between "recursive" > and "iterative" allocation. Looking at the dumps above, is it just a > question that SLOF is allocating the virtio device then the bridge at each > level, whereas PC bios is allocating the bridge then the virtio device? In fact, for each bridge, BIOS allocates first the inner pci-bridge and then the devices, so all pci-bridges are aligned to the same address. What I mean by "recursive" is inner pci-bridges are in the I/O address space of outter pci-bridges. with SLOF each pci-bridges I/O spaces are one after the other But I think in fact this doesn't save space: pci-bridges must be aligned on a 4KiB boundary for start and end address. So BIOS and SLOF are limited by the available I/O space. In our test case (meaning one virtio device on each pci-bridge), pseries can have 7 pci-bridges (it ignores them with 8) and pc can have 8 pci-bridges (BIOS doesn't start with 9). As you say in comment 11, I think one is directly allocated to the integrated legacy devices on the root bus. In conclusion, I think we cannot fix this BZ as we cannot change the size of the total I/O space or the needed alignment of the pci-bridge. > What I mean by "recursive" is inner pci-bridges are in the I/O address space of outter pci-bridges. with SLOF each pci-bridges I/O spaces are one after the other Uh.. no.. according to comment 10 on x86 the bridge IO spaces are nested as well (as they must be). It's just that for each layer inwards x86 is reducing the high address, whereas ppc is increasing the low address. So, yes we need 4kiB alignment, but for some reason ppc is consuming 8kiB per bridge whereas x86 is only consuming 4kiB. We ought to be able to achieve the second state by some means. (In reply to David Gibson from comment #17) > > What I mean by "recursive" is inner pci-bridges are in the I/O address space of outter pci-bridges. with SLOF each pci-bridges I/O spaces are one after the other > > Uh.. no.. according to comment 10 on x86 the bridge IO spaces are nested as > well (as they must be). It's just that for each layer inwards x86 is > reducing the high address, whereas ppc is increasing the low address. > > So, yes we need 4kiB alignment, but for some reason ppc is consuming 8kiB > per bridge whereas x86 is only consuming 4kiB. We ought to be able to > achieve the second state by some means. Yes, you're right, I was confusing because all pci-bridges are at the same address on x86. FWIW, here's a shorter command line to reproduce the issue (using a for-loop to generate the setup): sudo /usr/libexec/qemu-kvm -nographic -nodefaults -serial mon:stdio \ -m 8G -device pci-bridge,id=bridge1,chassis_nr=1,bus=pci.0 \ -drive file=main_image.qcow2,if=none,id=drv1,format=qcow2,cache=none \ -device virtio-blk-pci,drive=drv1,id=disk1,bus=bridge1,addr=0x1 \ `p=1 ; for ((i=2;i<=8;i++)) ; do echo -n " -device pci-bridge,id=bridge$i,chassis_nr=$i,bus=bridge$p,addr=0x2 -drive file=/tmp/storage$i.qcow2,if=none,id=drive$i,format=qcow2 -device virtio-blk-pci,drive=drive$i,id=disk$i,bus=bridge$i,addr=0x1"; p=$i ; done` ------- Comment From fnovak.com 2017-05-11 00:48 EDT------- RH1443433 - [ppc64le] - SLOF - Guest failed to boot up with 8 nested pci-bridge The problem has likely been introduced by this commit here: http://git.qemu.org/?p=SLOF.git;a=commitdiff;h=e53c2541 I think it should be possible to avoid the problem by reserving less than 4k for hotplugging ... I'll try to come up with a patch... This patch seems to fix the issue for me:
diff a/slof/fs/pci-scan.fs b/slof/fs/pci-scan.fs
--- a/slof/fs/pci-scan.fs
+++ b/slof/fs/pci-scan.fs
@@ -145,7 +145,8 @@ here 100 allot CONSTANT pci-device-vec
\ needed for scanning possible devices behind the bridge
: pci-bridge-set-io-base ( addr -- )
pci-next-io @ 1000 #aligned \ read the current Value and align to 4KB boundary
- dup 1000 + pci-next-io ! \ and write back with 4K for bridge
+ pci-next-io @ 800 + 1000 #aligned \ make sure we've got at least 2k for hotplugging
+ pci-next-io ! \ and write back
over 1C + rtas-config-l@ \ check if 32bit support
1 and IF \ IF 32 bit support
2dup 10 rshift \ | keep upper 16 bits
... need to more some more testing, but if that goes fine, I'll suggest this patch upstream.
Right. Reserving space for hotplugging seems a bit premature since hotplugging behind a bridge doesn't work yet anyway. Suggested patch upstream: https://lists.ozlabs.org/pipermail/slof/2017-May/001552.html Fix included in SLOF-20170303-4.git66d250e.el7 This bug is verified pass with same steps in comment 0 with SLOF-20170303-4.git66d250e.el7.noarch. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:2093 |