Bug 1322713

Summary: Guest Call Trace when system_reset on hmp launch guest with usb-storage device.
Product: Red Hat Enterprise Linux 7 Reporter: weliao <weliao>
Component: qemu-kvm-rhevAssignee: Gerd Hoffmann <kraxel>
Status: CLOSED ERRATA QA Contact: Xujun Ma <xuma>
Severity: high Docs Contact:
Priority: high    
Version: 7.3CC: chayang, huding, jen, juzhang, knoel, mrezanin, pbonzini, virt-maint, xfu
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: QEMU 2.6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-11-07 21:03:40 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
console info none

Description weliao 2016-03-31 08:04:04 UTC
Description of problem:
Launch guest with usb-storage, then reboot guest many times, guest Call Trace because No filesystem could mount root, tried:

Version-Release number of selected component (if applicable):
Host:
3.10.0-369.el7.x86_64
qemu-kvm-rhev-2.5.0-3.el7.x86_64
seabios-1.9.1-1.el7.x86_64
Guest:
3.10.0-369.el7.x86_64

How reproducible:
10%

Steps to Reproduce:
1.Launch guest with usb-storage.
/usr/libexec/qemu-kvm -name rhel7.3 -M pc -cpu SandyBridge \
-m 4G -smp 1,maxcpus=160  \
-boot order=d,menu=on \
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0 \
-boot menu=on   \
-netdev tap,id=tap1 -device virtio-net-pci,netdev=tap1,id=nic1,mac=52:55:00:5c:89:4d,bus=pci.0 \
-spice port=5900,disable-ticketing,  \
-monitor stdio -qmp tcp:0:5556,server,nowait \
-device usb-ehci,id=ehci -drive file=/home/RHEL-Server-7.3-64-virtio.qcow2,if=none,id=storage0,media=disk,cache=none,format=qcow2 \
-device usb-storage,bus=ehci.0,drive=storage0,id=storage0-0 \
-monitor unix:/tmp/monitor-unix,server,nowait \
-serial unix:/tmp/console,server,nowait
2.reboot guest via hmp
(qemu) system_reset
3.

Actual results:
Guest Call Trace and can't start.

Expected results:
Guest work normal

Additional info:
[    1.513380] usb 1-1: new high-speed USB device number 2 using ehci-pci
[    1.633283] usb 1-1: New USB device found, idVendor=46f4, idProduct=0001
[    1.634612] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    1.635662] usb 1-1: Product: QEMU USB HARDDRIVE
[    1.636453] usb 1-1: Manufacturer: QEMU
[    1.638239] usb 1-1: SerialNumber: 1-0000:00:04.0-1
[    2.055943] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8042/serio1/input/input2
[    2.057682] md: Waiting for all devices to be available before autodetect
[    2.059694] md: If you don't use raid, use raid=noautodetect
[    2.060771] md: Autodetecting RAID arrays.
[    2.061513] md: Scanned 0 and added 0 devices.
[    2.062275] md: autorun ...
[    2.062893] md: ... autorun DONE.
[    2.063577] List of all partitions:
[    2.064254] No filesystem could mount root, tried: 
[    2.065115] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[    2.066555] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10.0-369.el7.x86_64 #1
[    2.067896] Hardware name: Red Hat KVM, BIOS 1.9.1-1.el7 04/01/2014
[    2.068870]  ffffffff81864350 00000000c08718db ffff8801386efd60 ffffffff81647ab7
[    2.070419]  ffff8801386efde0 ffffffff81640bcc ffffffff00000010 ffff8801386efdf0
[    2.071968]  ffff8801386efd90 00000000c08718db 00000000c08718db ffff8801386efe00
[    2.073544] Call Trace:
[    2.074126]  [<ffffffff81647ab7>] dump_stack+0x19/0x1b
[    2.074966]  [<ffffffff81640bcc>] panic+0xd8/0x1e7
[    2.075769]  [<ffffffff81aaa5fa>] mount_block_root+0x2a1/0x2b0
[    2.076660]  [<ffffffff81aaa65c>] mount_root+0x53/0x56
[    2.077488]  [<ffffffff81aaa79b>] prepare_namespace+0x13c/0x174
[    2.078389]  [<ffffffff81aaa268>] kernel_init_freeable+0x1f0/0x217
[    2.079316]  [<ffffffff81aa99db>] ? initcall_blacklist+0xb0/0xb0
[    2.080231]  [<ffffffff816369e0>] ? rest_init+0x80/0x80
[    2.081066]  [<ffffffff816369ee>] kernel_init+0xe/0xf0
[    2.081899]  [<ffffffff81657e58>] ret_from_fork+0x58/0x90
[    2.082758]  [<ffffffff816369e0>] ? rest_init+0x80/0x80

Retest qemu-kvm-rhev-2.3.0-31.el7.x86_64.rpm. no this issue, so it's a regression bug.

Comment 1 weliao 2016-03-31 08:06:30 UTC
Created attachment 1142092 [details]
console info

Comment 5 Gerd Hoffmann 2016-04-06 09:43:59 UTC
[    0.602002] Unpacking initramfs...
[    1.087632] Initramfs unpacking failed: broken padding
[    1.088757] Freeing initrd memory: 57040k freed

Appearently loading the initrd failed.

Comment 6 Gerd Hoffmann 2016-04-07 05:52:57 UTC
Bisecting points to this:

commit 7070e085d490c396f9237c8f10bf8b6e69cd0066
Author: Paolo Bonzini <pbonzini>
Date:   Thu Jun 18 18:47:25 2015 +0200

    acpi: mark PMTIMER as unlocked
    
    Accessing QEMU_CLOCK_VIRTUAL is thread-safe.
    
    Signed-off-by: Paolo Bonzini <pbonzini>
    Message-Id: <1434646046-27150-9-git-send-email-pbonzini>

diff --git a/hw/acpi/core.c b/hw/acpi/core.c
index 0f201d8..fe6215a 100644
--- a/hw/acpi/core.c
+++ b/hw/acpi/core.c
@@ -528,6 +528,7 @@ void acpi_pm_tmr_init(ACPIREGS *ar, acpi_update_sci_fn update_sci,
     ar->tmr.timer = timer_new_ns(QEMU_CLOCK_VIRTUAL, acpi_pm_tmr_timer, ar);
     memory_region_init_io(&ar->tmr.io, memory_region_owner(parent),
                           &acpi_pm_tmr_ops, ar, "acpi-tmr", 4);
+    memory_region_clear_global_locking(&ar->tmr.io);
     memory_region_add_subregion(parent, 8, &ar->tmr.io);
 }
 
/me looks surprised.
But reverting it indeed fixes the problem.
And seabios actually uses pmtimer for timekeeping.

Note: root cause for the initrd issue noted in comment 5 is seabios running into problems with ehci -> io errors -> corrupted initrd.  Sometimes it doesn't boot at all, probably in case the io errors happen to hit the kernel not the initrd.

Paolo, any idea?

Comment 7 Gerd Hoffmann 2016-04-15 06:44:32 UTC
Revert submitted: https://patchwork.ozlabs.org/patch/610766/

Comment 9 weliao 2016-08-17 02:40:51 UTC
Verify this bug with following versions:
Host:
3.10.0-491.el7.x86_64
qemu-kvm-rhev-2.6.0-20.el7.x86_64
seabios-bin-1.9.1-4.el7.noarch
Guest:
3.10.0-478.el7.x86_64

Test step:
1.Launch guest with usb-storage.
# /usr/libexec/qemu-kvm -name rhel7.3 -M pc -cpu SandyBridge -m 4G -smp 1,maxcpus=160  -boot order=d,menu=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0 -boot menu=on   -netdev tap,id=tap1 -device virtio-net-pci,netdev=tap1,id=nic1,mac=52:55:00:5c:89:4d,bus=pci.0 -spice port=5900,disable-ticketing,  -monitor stdio -qmp tcp:0:5556,server,nowait -device usb-ehci,id=ehci -drive file=/home/RHEL-Server-7.3-64-virtio.qcow2,if=none,id=storage0,media=disk,cache=none,format=qcow2 -device usb-storage,bus=ehci.0,drive=storage0,id=storage0-0 -monitor unix:/tmp/monitor-unix,server,nowait -serial unix:/tmp/console,server,nowait
2.reboot guest via hmp
(qemu) system_reset

result:
guest work well, no Call Trace.

Tested 10 times all work well, so this bug fix well.

Comment 11 errata-xmlrpc 2016-11-07 21:03:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHBA-2016-2673.html