Bug 1261288
Summary: | 'Data Storage Exception [ fc36#### ]' when boot up guest with 10+ pci-ohci usb disks | ||
---|---|---|---|
Product: | Red Hat Enterprise Linux 7 | Reporter: | Gu Nini <ngu> |
Component: | SLOF | Assignee: | Thomas Huth <thuth> |
Status: | CLOSED ERRATA | QA Contact: | Virtualization Bugs <virt-bugs> |
Severity: | high | Docs Contact: | |
Priority: | medium | ||
Version: | 7.2 | CC: | knoel, michen, mrezanin, ngu, qzhang, thuth, virt-maint, xuhan, xuma, zhengtli |
Target Milestone: | rc | ||
Target Release: | --- | ||
Hardware: | ppc64 | ||
OS: | Unspecified | ||
Whiteboard: | |||
Fixed In Version: | qemu-slof-20160223 | Doc Type: | Bug Fix |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2016-11-04 04:31:38 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: | |||
Bug Depends On: | 1279954 | ||
Bug Blocks: |
Description
Gu Nini
2015-09-09 06:59:10 UTC
Besides, if boot up the guest with 38/39 usb disks, get following result: [root@ibm-p8-rhevm-10 home]# cat usbohci.sh #!/bin/bash CLI="/usr/libexec/qemu-kvm -name virtioblk-0828-le1 -machine pseries,accel=kvm,usb=off -m 2048M -realtime mlock=off -smp 152,sockets=1,cores=20,threads=8 -uuid 95346a10-1828-403a-a610-ac5a52a29415 -no-user-config -nodefaults -monitor stdio -rtc base=utc,clock=host -no-shutdown -boot strict=on -device spapr-vscsi,id=scsi0,reg=0x1000 -drive id=drive_image1,if=none,cache=none,snapshot=off,aio=native,format=qcow2,file=/home/virtioblk-0828-le1 -device virtio-blk-pci,id=image1,drive=drive_image1,bus=pci.0,addr=04,bootindex=1 -netdev tap,id=hostnet0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown -device spapr-vlan,netdev=hostnet0,id=net0,mac=52:54:00:c4:e7:15,reg=0x2000 -vnc 0:05 -msg timestamp=on -device VGA,id=video0,bus=pci.0,addr=0x8 -device nec-usb-xhci,id=xhci0 -device usb-kbd,id=input0,bus=xhci0.0 -device usb-mouse,id=input1,bus=xhci0.0 -device usb-tablet,id=input2,bus=xhci0.0 -chardev socket,id=serial1,path=/tmp/serial1,server,nowait,server,nowait -device spapr-vty,chardev=serial1,reg=0x20000000 -device pci-ohci,id=ohci0" while [ ${i:=0} -lt ${1:-1} ] CLI="$CLI -drive file=/home/disk/disk$i.qcow2,if=none,id=drive-usb-0-$i,media=disk,format=qcow2,cache=none,aio=native" CLI="$CLI -device usb-storage,drive=drive-usb-0-$i,id=usb-0-$i,removable=on,bus=ohci0.0" i=$((i+1)) done [root@ibm-p8-rhevm-10 home]# ./usbohci.sh 39 QEMU 2.3.0 monitor - type 'help' for more information (qemu) 2015-09-09T08:50:18.814620Z qemu-kvm: -device usb-storage,drive=drive-usb-0-38,id=usb-0-38,removable=on,bus=ehci1.0: tried to attach usb device QEMU USB MSD to a bus with no free ports 2015-09-09T08:50:18.814675Z qemu-kvm: -device usb-storage,drive=drive-usb-0-38,id=usb-0-38,removable=on,bus=ehci1.0: Device 'usb-storage' could not be initialized [root@ibm-p8-rhevm-10 home]# [root@ibm-p8-rhevm-10 home]# [root@ibm-p8-rhevm-10 home]# ./usbohci.sh 38 QEMU 2.3.0 monitor - type 'help' for more information (qemu) (qemu) (qemu) (qemu) info usb Device 0.0, Port 1, Speed 480 Mb/s, Product QEMU USB Keyboard Device 0.0, Port 2, Speed 480 Mb/s, Product QEMU USB Mouse Device 0.0, Port 3, Speed 480 Mb/s, Product QEMU USB Tablet Device 1.1, Port 1, Speed 12 Mb/s, Product QEMU USB MSD Device 1.2, Port 2, Speed 12 Mb/s, Product QEMU USB MSD Device 1.3, Port 3, Speed 12 Mb/s, Product QEMU USB Hub Device 1.4, Port 3.1, Speed 12 Mb/s, Product QEMU USB MSD Device 1.5, Port 3.2, Speed 12 Mb/s, Product QEMU USB MSD Device 1.6, Port 3.3, Speed 12 Mb/s, Product QEMU USB MSD Device 1.7, Port 3.4, Speed 12 Mb/s, Product QEMU USB MSD Device 1.8, Port 3.5, Speed 12 Mb/s, Product QEMU USB MSD Device 1.9, Port 3.6, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.7, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8, Speed 12 Mb/s, Product QEMU USB Hub Device 1.0, Port 3.8.1, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8.2, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8.3, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8.4, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8.5, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8.6, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8.7, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8.8, Speed 12 Mb/s, Product QEMU USB Hub Device 1.0, Port 3.8.8.1, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8.8.2, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8.8.3, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8.8.4, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8.8.5, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8.8.6, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8.8.7, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8.8.8, Speed 12 Mb/s, Product QEMU USB Hub Device 1.0, Port 3.8.8.8.1, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8.8.8.2, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8.8.8.3, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8.8.8.4, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8.8.8.5, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8.8.8.6, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8.8.8.7, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8.8.8.8, Speed 12 Mb/s, Product QEMU USB Hub Device 1.0, Port 3.8.8.8.8.1, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8.8.8.8.2, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8.8.8.8.3, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8.8.8.8.4, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8.8.8.8.5, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8.8.8.8.6, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8.8.8.8.7, Speed 12 Mb/s, Product QEMU USB MSD Device 1.0, Port 3.8.8.8.8.8, Speed 12 Mb/s, Product QEMU USB MSD -------> **the 8th device on the 5th hub is the additional one** (qemu) In conclusion, 37(8*5-5+3-1) usb devices should be the maximum, while above test result shows 38 is supported at present. 9 is probably more than enough USD disks for RHEL 7.2 guests. Move to 7.3. (In reply to Gu Nini from comment #0) > Description of problem: > If boot up guest with 10 or more usb disks on the same pci-ohci usb > controllers, the guest could not boot up and met 'Data Storage Exception [ > fc36#### ]' > > Version-Release number of selected component (if applicable): > Host kernel: 3.10.0-306.0.1.el7.ppc64le > Guest kernel: 3.10.0-306.0.1.el7.ppc64le > Qemu-kvm-rhev: qemu-kvm-rhev-2.3.0-22.el7.ppc64le > > How reproducible: > 100% > > Steps to Reproduce: > 1. Boot up guest with 10 or more usb disks on the same pci-ohci usb > controller: > > /usr/libexec/qemu-kvm -name virtioblk-0828-le1 -machine > pseries,accel=kvm,usb=off **-m 2048M** -realtime mlock=off -smp ...... If increase guest memory **-m 2048M** to 4096M, the exception in the bug disappears; however there is still following 'Cannot open file : dev-storage.fs' OHCI controller initialization related info during the guest booting up process; BTW, for xhci usb controller, there is no the bug and controller initialization related problems when boot up guest with the maximum 39 usb disks similarly. SLOF ********************************************************************** QEMU Starting Build Date = Sep 18 2015 06:25:39 FW Version = mockbuild@ release 20150313 Press "s" to enter Open Firmware. Populating /vdevice methods Populating /vdevice/v-scsi@1000 SCSI: Looking for devices 8000000000000000 CD-ROM : "QEMU QEMU CD-ROM 2.3." Populating /vdevice/vty@20000000 Populating /vdevice/nvram@71000000 Populating /pci@800000020000000 00 4800 (D) : 1af4 1000 virtio [ net ] 00 0800 (D) : 106b 003f serial bus [ usb-ohci ] 00 0000 (D) : 1af4 1001 virtio [ block ] Scanning USB OHCI: initializing USB Storage SCSI: Looking for devices 101000000000000 DISK : "QEMU QEMU HARDDISK 2.3." USB Storage SCSI: Looking for devices 102000000000000 DISK : "QEMU QEMU HARDDISK 2.3." USB HUB USB Storage SCSI: Looking for devices 101000000000000 DISK : "QEMU QEMU HARDDISK 2.3." USB Storage SCSI: Looking for devices 102000000000000 DISK : "QEMU QEMU HARDDISK 2.3." USB Storage SCSI: Looking for devices 103000000000000 DISK : "QEMU QEMU HARDDISK 2.3." USB Storage SCSI: Looking for devices 104000000000000 DISK : "QEMU QEMU HARDDISK 2.3." USB Storage SCSI: Looking for devices 105000000000000 DISK : "QEMU QEMU HARDDISK 2.3." USB Storage SCSI: Looking for devices 106000000000000 DISK : "QEMU QEMU HARDDISK 2.3." USB Storage SCSI: Looking for devices 107000000000000 DISK : "QEMU QEMU HARDDISK 2.3." USB HUB Cannot open file : dev-storage.fs Cannot open file : dev-storage.fs Cannot open file : dev-storage.fs Cannot open file : dev-storage.fs Cannot open file : dev-storage.fs Cannot open file : dev-storage.fs Cannot open file : dev-storage.fs Cannot open file : dev-hub.fs claim failed! Using default console: /vdevice/vty@20000000 Welcome to Open Firmware I think this is probably a SLOF bug, just failing to allocate enough space internally to track that many USB devices. I'm moving to the SLOF component, we can move it back if it does turn out to be a qemu bug. (In reply to Gu Nini from comment #2) > Besides, if boot up the guest with 38/39 usb disks, get following result: ... > In conclusion, 37(8*5-5+3-1) usb devices should be the maximum, while above > test result shows 38 is supported at present. Not sure if I've understood you correctly here ... do you mean QEMU should abort after 37 devices already? On the last bus level, you can plug 8 devices instead of seven since there is no slot used for a hub anymore, so as far as I can tell the calculation should be 8*5-4+3-1 = 38 ... and that's what you get? Or did I miss something? The problem looks like a stack-overflow in SLOF to me, most likely caused by the recursion which is done while scanning the hubs. I'll have a look whether we can save some stack space in the USB code of SLOF, or whether increasing the stack globally is feasible instead... (In reply to Thomas Huth from comment #6) > (In reply to Gu Nini from comment #2) > > Besides, if boot up the guest with 38/39 usb disks, get following result: > ... > > In conclusion, 37(8*5-5+3-1) usb devices should be the maximum, while above > > test result shows 38 is supported at present. > > Not sure if I've understood you correctly here ... do you mean QEMU should > abort after 37 devices already? On the last bus level, you can plug 8 > devices instead of seven since there is no slot used for a hub anymore, so > as far as I can tell the calculation should be 8*5-4+3-1 = 38 ... and that's > what you get? Or did I miss something? Sorry, it's my fault, 38 is correct, please neglect commnet 2. Thanks for the notification. FWIW, an even simpler reproducer is to use USB mice (then you don't need so much disk images): /usr/libexec/qemu-kvm -nographic -vga none -device pci-ohci,id=ohci0 `for ((i=0;i<38;i++)); do echo -n "-device usb-mouse " ; done` Anyway, I've identified now a couple of spots in the code that contribute to the heavy stack usage and thus the stack overflow. I'm working on fixing these now... I've posted now my patches that fix this issue upstream: https://lists.ozlabs.org/pipermail/slof/2015-November/000383.html Patches have been accepted by upstream: https://github.com/aik/SLOF/commit/3d5e62eea27cab3deaa1f9399dd619ad1638fa89 https://github.com/aik/SLOF/commit/edd582383f9800e61b48669f7515e5d334ef961b https://github.com/aik/SLOF/commit/f0d251a0775572ebce8566c16e4482455b9efd84 https://github.com/aik/SLOF/commit/2ef6d1e52d5b6f3a8198c1c39a55eeff2981b282 https://github.com/aik/SLOF/commit/baa834884c90c243c1467c380ac3b4a51b42b967 https://github.com/aik/SLOF/commit/271fd45605c119de0b600017df8ff10137a749b8 We'll get the fixes when rebasing SLOF (see bz1279954) This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune with any questions Fix included in SLOF-20160223-1.gitdbbfda4.el7. I can reproduce the problem with SLOF-20150313-5.gitc89b0df.el7.noarch. follows by the reproducer Comment #9 [root@ibm-p8-rhevm-06 test]# /usr/libexec/qemu-kvm -nographic -vga none -device pci-ohci,id=ohci0 `for ((i=0;i<38;i++)); do echo -n "-device usb-mouse " ; done` SLOF ********************************************************************** QEMU Starting Build Date = Sep 18 2015 06:25:39 FW Version = mockbuild@ release 20150313 Press "s" to enter Open Firmware. Populating /vdevice methods Populating /vdevice/vty@71000000 Populating /vdevice/nvram@71000001 Populating /vdevice/l-lan@71000002 Populating /vdevice/v-scsi@71000003 SCSI: Looking for devices 8200000000000000 CD-ROM : "QEMU QEMU CD-ROM 2.5+" Populating /pci@800000020000000 00 0000 (D) : 106b 003f serial bus [ usb-ohci ] No NVRAM common partition, re-initializing... Scanning USB OHCI: initializing USB mouse USB mouse USB HUB USB mouse USB mouse USB mouse USB mouse USB mouse USB mouse USB mouse USB HUB ( 300 ) Data Storage Exception [ 3c1a5810 ] R0 .. R7 R8 .. R15 R16 .. R23 R24 .. R31 000000001dbe060c 000000003c1a57f0 0000000000000000 000000001e505f20 000000001e45b1a0 000000001dbe5fa4 0000000000000000 0000000000000006 000000001dbff278 000000001e5637d0 0000000000000000 000000001dbf6500 000000001e5637d0 000000003c1a5808 000000001dbe0dac 000000001e45bb60 000000003c1a57f0 0000000000000001 000000001dbfd5f0 000000001dbf8830 000000003c1a5810 6465762d6d6f7573 0000000000000044 000000000000000b 000000001dbe5f84 000000001e5637dc 000000001dbf7e80 000000001dc42038 000000001dc42028 000000001e505e18 000000001dbe17a0 000000001dbe3c74 CR / XER LR / CTR SRR0 / SRR1 DAR / DSISR 84000084 000000001dbe5f14 000000001dbe5f60 000000003c1a5810 0000000020000000 000000001dbe5f84 8000000000001000 40000000 5 > After I upgrade the slof version to SLOF-20160223-1.gitdbbfda4. Run the producer again. [root@ibm-p8-rhevm-06 test]# /usr/libexec/qemu-kvm -nographic -vga none -device pci-ohci,id=ohci0 `for ((i=0;i<38;i++)); do echo -n "-device usb-mouse " ; done` SLOF ********************************************************************** QEMU Starting Build Date = May 6 2016 07:26:27 FW Version = mockbuild@ release 20160223 Press "s" to enter Open Firmware. Populating /vdevice methods Populating /vdevice/vty@71000000 Populating /vdevice/nvram@71000001 Populating /vdevice/l-lan@71000002 Populating /vdevice/v-scsi@71000003 SCSI: Looking for devices 8200000000000000 CD-ROM : "QEMU QEMU CD-ROM 2.5+" Populating /pci@800000020000000 00 0000 (D) : 106b 003f serial bus [ usb-ohci ] No NVRAM common partition, re-initializing... Scanning USB OHCI: initializing USB mouse USB mouse USB HUB USB mouse USB mouse USB mouse USB mouse USB mouse USB mouse USB mouse USB HUB USB mouse USB mouse USB mouse USB mouse USB mouse USB mouse USB mouse USB HUB USB mouse USB mouse USB mouse USB mouse USB mouse USB mouse USB mouse USB HUB USB mouse USB mouse USB mouse USB mouse USB mouse USB mouse USB mouse USB HUB USB mouse USB mouse USB mouse USB mouse USB mouse USB mouse USB mouse USB mouse Using default console: /vdevice/vty@71000000 Welcome to Open Firmware Copyright (c) 2004, 2011 IBM Corporation All rights reserved. This program and the accompanying materials are made available under the terms of the BSD License available at http://www.opensource.org/licenses/bsd-license.php Trying to load: from: disk ... E3405: No such device Trying to load: from: /vdevice/v-scsi@71000003/disk@8200000000000000 ... No medium ! E3405: No such device Trying to load: from: /vdevice/l-lan@71000002 ... Initializing NIC Reading MAC address from device: 52:54:00:12:34:56 Requesting information via DHCP: done Using IPv4 address: 10.0.2.15 Requesting file "" via TFTP from 10.0.2.2 Receiving data: 0 KBytes E3010 (net) TFTP access violation E3407: Load failed ..`. .. ....... .. ...... ....... ..`...`''.`'. .''``````..''. .`''```''`. `''`````` .`` .:' ': `''..... .''. ''` .''..''....... ``.':.';. ``````''`.''. .''. ''``''`````'` ``.':':` .....`''.`'`...... `'`.....`''.`'` .`.`'`` .'`'`````. ``'''''' ``''`'''`. `'` Type 'boot' and press return to continue booting the system. Type 'reset-all' and press return to reboot the system. Ready! 0 > No such problems any more . so this bug could be marked verified. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHEA-2016-2355.html |