Bug 923653
| Summary: | KVM internal error. Suberror: 3 when doing system_reset in monitor | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | xhan | ||||||||||||
| Component: | qemu-kvm | Assignee: | Radim Krčmář <rkrcmar> | ||||||||||||
| Status: | CLOSED INSUFFICIENT_DATA | QA Contact: | Virtualization Bugs <virt-bugs> | ||||||||||||
| Severity: | medium | Docs Contact: | |||||||||||||
| Priority: | medium | ||||||||||||||
| Version: | 7.0 | CC: | drjones, hhuang, huding, juzhang, knoel, michen, pbonzini, qzhang, rbalakri, s.adam, shuang, virt-maint, xutian | ||||||||||||
| Target Milestone: | rc | ||||||||||||||
| Target Release: | --- | ||||||||||||||
| Hardware: | Unspecified | ||||||||||||||
| OS: | Unspecified | ||||||||||||||
| Whiteboard: | |||||||||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||||||||
| Doc Text: | Story Points: | --- | |||||||||||||
| Clone Of: | |||||||||||||||
| : | 1002794 (view as bug list) | Environment: | |||||||||||||
| Last Closed: | 2016-08-31 01:38:19 UTC | Type: | Bug | ||||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||||
| Documentation: | --- | CRM: | |||||||||||||
| Verified Versions: | Category: | --- | |||||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||
| Embargoed: | |||||||||||||||
| Attachments: |
|
||||||||||||||
Created attachment 713163 [details]
debug info
Guest=RHEL.6.4.64
Cmd:
/home/autotest/autotest-devel/client/tests/kvm/qemu
-S
-name 'vm1'
-nodefaults
-chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20130319-232118-nwJe57cS,server,nowait
-mon chardev=qmp_id_qmpmonitor1,mode=control
-chardev socket,id=serial_id_serial1,path=/tmp/serial-serial1-20130319-232118-nwJe57cS,server,nowait
-device isa-serial,chardev=serial_id_serial1
-chardev socket,id=seabioslog_id_20130319-232118-nwJe57cS,path=/tmp/seabios-20130319-232118-nwJe57cS,server,nowa
-device isa-debugcon,chardev=seabioslog_id_20130319-232118-nwJe57cS,iobase=0x402
-device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=0x4
-drive file='/home/autotest/autotest-devel/client/tests/kvm/images/RHEL-Server-6.4-64-virtio.qcow2',if=none,id=drive-virtio-disk1,media=disk,cache=unsafe,boot=off,snapshot=off,format=qcow2,aio=native
afe,boot=off,snapshot=off,format=qcow2,aio=native
-device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1
-device virtio-net-pci,netdev=idr3YiKW,mac=9a:ff:00:01:02:03,bus=pci.0,addr=0x3,id='id4WXlml'
-netdev tap,id=idr3YiKW,vhost=on,fd=23
-m 4096
-smp 4,maxcpus=4,cores=2,threads=1,sockets=2
-cpu 'SandyBridge'
-M pc
-device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1
-vnc :0
-vga cirrus
-rtc base=utc,clock=host,driftfix=slew
-boot order=cdn,once=c,menu=off
-no-kvm-pit-reinjection
-enable-kvm
Created attachment 716292 [details]
host cpu info
reproduced this bug on kernel-3.9.0-0.rc6.51.el7.x86_64(host), qemu-kvm-1.4.0-2.el7.x86_64;
steps below:
/usr/libexec/qemu-kvm -name RHEL6.4 \
-M pc \
-enable-kvm \
-m 4096 -smp 4,sockets=4,cores=1,threads=1 \
-uuid 34f07c2c-0290-2c5a-de2c-711eb2a3266f \
-nodefaults \
-rtc base=utc \
-boot order=dc,menu=on \
-drive file=/home/kvm_autotest_root/images/RHEL-Server-7.0-64-virtio.qcow2,if=none,id=virtio-disk0,format=qcow2,serial=0x101213,cache=unsafe,aio=native,werror=stop,rerror=stop \
-device ide-hd,drive=virtio-disk0 \
-vga qxl \
-global qxl-vga.ram_size=67108864 \
-global qxl-vga.vram_size=67108864 \
-monitor stdio \
-spice port=5909,addr=10.66.5.96,disable-ticketing \
-chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20130428-132443-r0PcFHRY,server,nowait \
-mon chardev=qmp_id_qmpmonitor1,mode=control \
-chardev socket,id=serial_id_serial1,path=/tmp/serial-serial1-20130428-132443-r0PcFHRY,server,nowait \
-device isa-serial,chardev=serial_id_serial1 \
-chardev socket,id=seabioslog_id_20130428-132443-r0PcFHRY,path=/tmp/seabios-20130428-132443-r0PcFHRY,server,nowait \
-device isa-debugcon,chardev=seabioslog_id_20130428-132443-r0PcFHRY,iobase=0x402
2. run below script on host:
#/bin/bash
set -x
header='{"execute": "qmp_capabilities"}\n'
echo -e $header'{"execute": "system_reset"}' > /tmp/system_reset
while :;do
`sleep $((RANDOM % 4 + 13))`
nc -U /tmp/monitor-qmpmonitor1-20130428-132443-r0PcFHRY < /tmp/system_reset
done
3.wait qemu report error
result: KVM internal error. Suberror: 1 emulation failure
detials please see attachment qemu-error.log
Created attachment 741075 [details]
qemu-error.log
Created attachment 741077 [details]
qemu-error.log
I've been trying to reproduce this on a i5-4670T with both host and guest running kernel-3.9.0-0.55.el7.x86_64, and with qemu-kvm-1.4.0-3.el7.x86_64. The guest has been resetting over and over most the day, but still no luck. Unlike suberror 1, suberror 3 gives a bit more information KVM internal error. Suberror: 3 extra data[0]: 80000b0e extra data[1]: 31 extra data[0] tells that the the event vm-exit occured because of a page fault. extra data[1] contains EPT misconfiguration exit reason. EPT misconfiguration is used for MMIO, and we even have an explanation why the internal error occured /* * Note: * Do not try to fix EXIT_REASON_EPT_MISCONFIG if it caused by * delivery event since it indicates guest is accessing MMIO. * The vm-exit can be triggered again after return to guest that * will cause infinite loop. */ We might be "accessing MMIO" by accident as we had suberror 1 too ... RIP+CS is suspicious and you don't see zeroed registers in this context, so a race comes into mind. Reproducer would help to get an idea. Please preserve the reproducer and reopen this bz if you happen to hit it again. |
Created attachment 713151 [details] screendump Description of problem: Reset guest by monitor many times cause kvm internal error. Suberror 1 and Suberror 3. Version-Release number of selected component (if applicable): #uname -r 3.8.0-0.40.el7.x86_64 # rpm -qa | grep qemu-kvm qemu-kvm-debuginfo-1.4.0-1.el7.x86_64 qemu-kvm-common-1.4.0-1.el7.x86_64 qemu-kvm-1.4.0-1.el7.x86_64 qemu-kvm-tools-1.4.0-1.el7.x86_64 How reproducible: 1/4 Steps to Reproduce: 1. Boot guest 2. Reset guest by monitor many times Using the script of autotest to reproduce this bug - system_reset_bootable Actual results: Reset the system by monitor cmd after 10s [qemu output] KVM internal error. Suberror: 1 [qemu output] emulation failure [qemu output] RAX=0000000000000000 RBX=0000000000000000 RCX=0000000000000000 RDX=00000000000206a1 [qemu output] RSI=0000000000000000 RDI=0000000000000000 RBP=0000000000000000 RSP=0000000000000000 [qemu output] R8 =0000000000000000 R9 =0000000000000000 R10=0000000000000000 R11=0000000000000000 [qemu output] R12=0000000000000000 R13=0000000000000000 R14=0000000000000000 R15=0000000000000000 [qemu output] RIP=000000000000fff0 RFL=00010002 [-------] CPL=3 II=0 A20=1 SMM=0 HLT=0 [qemu output] ES =0000 0000000000000000 ffffffff 00c00000 [qemu output] CS =0033 0000000000000000 ffffffff 00a0fb00 DPL=3 CS64 [-RA] [qemu output] SS =002b 0000000000000000 ffffffff 00c0f300 DPL=3 DS [-WA] [qemu output] DS =0000 0000000000000000 ffffffff 00c00000 [qemu output] FS =0000 00007fa00e014700 ffffffff 00c00000 [qemu output] GS =0000 0000000000000000 ffffffff 00c00000 [qemu output] LDT=0000 0000000000000000 ffffffff 00c00000 [qemu output] TR =0040 ffff880028294280 00002087 00008b00 DPL=0 TSS64-busy [qemu output] GDT= ffff880028284000 0000007f [qemu output] IDT= ffffffff81dde000 00000fff [qemu output] CR0=80050033 CR2=000000000000fff0 CR3=000000011c614000 CR4=000407e0 [qemu output] DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 [qemu output] DR6=00000000ffff0ff0 DR7=0000000000000400 [qemu output] EFER=0000000000000d01 [qemu output] Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??> ?? ?? ?? ?? ?? ? ? ?? ?? ?? ?? ?? [qemu output] KVM internal error. Suberror: 3 [qemu output] extra data[0]: 80000b0e [qemu output] extra data[1]: 31 [qemu output] RAX=0000000000000000 RBX=0000000000000000 RCX=0000000000000000 RDX=00000000000206a1 [qemu output] RSI=0000000000000000 RDI=0000000000000000 RBP=0000000000000000 RSP=0000000000000000 [qemu output] R8 =0000000000000000 R9 =0000000000000000 R10=0000000000000000 R11=0000000000000000 [qemu output] R12=0000000000000000 R13=0000000000000000 R14=0000000000000000 R15=0000000000000000 [qemu output] RIP=000000000000fff0 RFL=00010002 [-------] CPL=3 II=0 A20=1 SMM=0 HLT=0 [qemu output] ES =0000 0000000000000000 ffffffff 00c00000 [qemu output] CS =0033 0000000000000000 ffffffff 00a0fb00 DPL=3 CS64 [-RA] [qemu output] SS =002b 0000000000000000 ffffffff 00c0f300 DPL=3 DS [-WA] [qemu output] DS =0000 0000000000000000 ffffffff 00c00000 [qemu output] FS =0000 00007fa00e014700 ffffffff 00c00000 [qemu output] GS =0000 0000000000000000 ffffffff 00c00000 [qemu output] LDT=0000 0000000000000000 ffffffff 00c00000 [qemu output] TR =0040 ffff880028294280 00002087 00008b00 DPL=0 TSS64-busy [qemu output] GDT= ffff880028284000 0000007f [qemu output] IDT= ffffffff81dde000 00000fff [qemu output] CR0=80050033 CR2=000000000000fff0 CR3=000000011c614000 CR4=000407e0 [qemu output] DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 [qemu output] DR6=00000000ffff0ff0 DR7=0000000000000400 [qemu output] EFER=0000000000000d01 [qemu output] Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??> ?? ?? ?? ?? ?? Expected results: Guest should work well. Additional info: