Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 923653

Summary: KVM internal error. Suberror: 3 when doing system_reset in monitor
Product: Red Hat Enterprise Linux 7 Reporter: xhan
Component: qemu-kvmAssignee: Radim Krčmář <rkrcmar>
Status: CLOSED INSUFFICIENT_DATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 7.0CC: drjones, hhuang, huding, juzhang, knoel, michen, pbonzini, qzhang, rbalakri, s.adam, shuang, virt-maint, xutian
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1002794 (view as bug list) Environment:
Last Closed: 2016-08-31 01:38:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
screendump
none
debug info
none
host cpu info
none
qemu-error.log
none
qemu-error.log none

Description xhan 2013-03-20 09:48:49 UTC
Created attachment 713151 [details]
screendump

Description of problem:

Reset guest by monitor many times cause kvm internal error. Suberror 1 and Suberror 3.

Version-Release number of selected component (if applicable):
    #uname -r 
    3.8.0-0.40.el7.x86_64

    # rpm -qa | grep qemu-kvm
    qemu-kvm-debuginfo-1.4.0-1.el7.x86_64
    qemu-kvm-common-1.4.0-1.el7.x86_64
    qemu-kvm-1.4.0-1.el7.x86_64
    qemu-kvm-tools-1.4.0-1.el7.x86_64



How reproducible:
1/4

Steps to Reproduce:
1. Boot guest 
2. Reset guest by monitor many times

Using the script of autotest to reproduce this bug 
   - system_reset_bootable

  
Actual results:

Reset the system by monitor cmd after 10s
[qemu output] KVM internal error. Suberror: 1
[qemu output] emulation failure
[qemu output] RAX=0000000000000000 RBX=0000000000000000 RCX=0000000000000000 RDX=00000000000206a1
[qemu output] RSI=0000000000000000 RDI=0000000000000000 RBP=0000000000000000 RSP=0000000000000000
[qemu output] R8 =0000000000000000 R9 =0000000000000000 R10=0000000000000000 R11=0000000000000000
[qemu output] R12=0000000000000000 R13=0000000000000000 R14=0000000000000000 R15=0000000000000000
[qemu output] RIP=000000000000fff0 RFL=00010002 [-------] CPL=3 II=0 A20=1 SMM=0 HLT=0
[qemu output] ES =0000 0000000000000000 ffffffff 00c00000
[qemu output] CS =0033 0000000000000000 ffffffff 00a0fb00 DPL=3 CS64 [-RA]
[qemu output] SS =002b 0000000000000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
[qemu output] DS =0000 0000000000000000 ffffffff 00c00000
[qemu output] FS =0000 00007fa00e014700 ffffffff 00c00000
[qemu output] GS =0000 0000000000000000 ffffffff 00c00000
[qemu output] LDT=0000 0000000000000000 ffffffff 00c00000
[qemu output] TR =0040 ffff880028294280 00002087 00008b00 DPL=0 TSS64-busy
[qemu output] GDT=     ffff880028284000 0000007f
[qemu output] IDT=     ffffffff81dde000 00000fff
[qemu output] CR0=80050033 CR2=000000000000fff0 CR3=000000011c614000 CR4=000407e0
[qemu output] DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
[qemu output] DR6=00000000ffff0ff0 DR7=0000000000000400
[qemu output] EFER=0000000000000d01
[qemu output] Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??> ?? ?? ?? ?? ?? ?
? ?? ?? ?? ?? ??
[qemu output] KVM internal error. Suberror: 3
[qemu output] extra data[0]: 80000b0e
[qemu output] extra data[1]: 31
[qemu output] RAX=0000000000000000 RBX=0000000000000000 RCX=0000000000000000 RDX=00000000000206a1
[qemu output] RSI=0000000000000000 RDI=0000000000000000 RBP=0000000000000000 RSP=0000000000000000
[qemu output] R8 =0000000000000000 R9 =0000000000000000 R10=0000000000000000 R11=0000000000000000
[qemu output] R12=0000000000000000 R13=0000000000000000 R14=0000000000000000 R15=0000000000000000
[qemu output] RIP=000000000000fff0 RFL=00010002 [-------] CPL=3 II=0 A20=1 SMM=0 HLT=0
[qemu output] ES =0000 0000000000000000 ffffffff 00c00000
[qemu output] CS =0033 0000000000000000 ffffffff 00a0fb00 DPL=3 CS64 [-RA]
[qemu output] SS =002b 0000000000000000 ffffffff 00c0f300 DPL=3 DS   [-WA]
[qemu output] DS =0000 0000000000000000 ffffffff 00c00000
[qemu output] FS =0000 00007fa00e014700 ffffffff 00c00000
[qemu output] GS =0000 0000000000000000 ffffffff 00c00000
[qemu output] LDT=0000 0000000000000000 ffffffff 00c00000
[qemu output] TR =0040 ffff880028294280 00002087 00008b00 DPL=0 TSS64-busy
[qemu output] GDT=     ffff880028284000 0000007f
[qemu output] IDT=     ffffffff81dde000 00000fff
[qemu output] CR0=80050033 CR2=000000000000fff0 CR3=000000011c614000 CR4=000407e0
[qemu output] DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
[qemu output] DR6=00000000ffff0ff0 DR7=0000000000000400
[qemu output] EFER=0000000000000d01
[qemu output] Code=?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? <??> ?? ?? ?? ?? ??

Expected results:
Guest should work well.

Additional info:

Comment 1 xhan 2013-03-20 09:52:09 UTC
Created attachment 713163 [details]
debug info

Comment 3 xhan 2013-03-20 10:17:23 UTC
Guest=RHEL.6.4.64

Cmd:
/home/autotest/autotest-devel/client/tests/kvm/qemu 
    -S 
    -name 'vm1' 
    -nodefaults 
    -chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20130319-232118-nwJe57cS,server,nowait 
    -mon chardev=qmp_id_qmpmonitor1,mode=control 
    -chardev socket,id=serial_id_serial1,path=/tmp/serial-serial1-20130319-232118-nwJe57cS,server,nowait 
    -device isa-serial,chardev=serial_id_serial1 
    -chardev socket,id=seabioslog_id_20130319-232118-nwJe57cS,path=/tmp/seabios-20130319-232118-nwJe57cS,server,nowa

    -device isa-debugcon,chardev=seabioslog_id_20130319-232118-nwJe57cS,iobase=0x402 
    -device ich9-usb-uhci1,id=usb1,bus=pci.0,addr=0x4 
    -drive file='/home/autotest/autotest-devel/client/tests/kvm/images/RHEL-Server-6.4-64-virtio.qcow2',if=none,id=drive-virtio-disk1,media=disk,cache=unsafe,boot=off,snapshot=off,format=qcow2,aio=native
afe,boot=off,snapshot=off,format=qcow2,aio=native 
    -device virtio-blk-pci,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 
    -device virtio-net-pci,netdev=idr3YiKW,mac=9a:ff:00:01:02:03,bus=pci.0,addr=0x3,id='id4WXlml' 
    -netdev tap,id=idr3YiKW,vhost=on,fd=23 
    -m 4096 
    -smp 4,maxcpus=4,cores=2,threads=1,sockets=2 
    -cpu 'SandyBridge' 
    -M pc 
    -device usb-tablet,id=usb-tablet1,bus=usb1.0,port=1 
    -vnc :0 
    -vga cirrus 
    -rtc base=utc,clock=host,driftfix=slew  
    -boot order=cdn,once=c,menu=off   
    -no-kvm-pit-reinjection 
    -enable-kvm

Comment 5 xhan 2013-03-26 02:21:29 UTC
Created attachment 716292 [details]
host cpu info

Comment 6 Xu Tian 2013-04-28 07:43:36 UTC
reproduced this bug on kernel-3.9.0-0.rc6.51.el7.x86_64(host), qemu-kvm-1.4.0-2.el7.x86_64;

steps below:

/usr/libexec/qemu-kvm -name RHEL6.4 \
 -M pc \
-enable-kvm \
-m 4096 -smp 4,sockets=4,cores=1,threads=1 \
-uuid 34f07c2c-0290-2c5a-de2c-711eb2a3266f  \
-nodefaults \
-rtc base=utc \
-boot order=dc,menu=on \
-drive file=/home/kvm_autotest_root/images/RHEL-Server-7.0-64-virtio.qcow2,if=none,id=virtio-disk0,format=qcow2,serial=0x101213,cache=unsafe,aio=native,werror=stop,rerror=stop \
 -device ide-hd,drive=virtio-disk0 \
-vga qxl \
-global qxl-vga.ram_size=67108864 \
-global qxl-vga.vram_size=67108864 \
-monitor stdio \
-spice port=5909,addr=10.66.5.96,disable-ticketing  \
-chardev socket,id=qmp_id_qmpmonitor1,path=/tmp/monitor-qmpmonitor1-20130428-132443-r0PcFHRY,server,nowait \
-mon chardev=qmp_id_qmpmonitor1,mode=control \
-chardev socket,id=serial_id_serial1,path=/tmp/serial-serial1-20130428-132443-r0PcFHRY,server,nowait \
-device isa-serial,chardev=serial_id_serial1 \
-chardev socket,id=seabioslog_id_20130428-132443-r0PcFHRY,path=/tmp/seabios-20130428-132443-r0PcFHRY,server,nowait \
-device isa-debugcon,chardev=seabioslog_id_20130428-132443-r0PcFHRY,iobase=0x402 

2. run below script on host:
#/bin/bash
set -x
header='{"execute": "qmp_capabilities"}\n'
echo -e $header'{"execute": "system_reset"}' > /tmp/system_reset
while  :;do 
`sleep $((RANDOM % 4 + 13))`
nc -U /tmp/monitor-qmpmonitor1-20130428-132443-r0PcFHRY < /tmp/system_reset
done

3.wait qemu report error

result: KVM internal error. Suberror: 1 emulation failure

detials please see attachment qemu-error.log

Comment 7 Xu Tian 2013-04-28 07:44:25 UTC
Created attachment 741075 [details]
qemu-error.log

Comment 8 Xu Tian 2013-04-28 07:46:45 UTC
Created attachment 741077 [details]
qemu-error.log

Comment 9 Andrew Jones 2013-05-03 14:29:50 UTC
I've been trying to reproduce this on a i5-4670T with both host and guest running kernel-3.9.0-0.55.el7.x86_64, and with qemu-kvm-1.4.0-3.el7.x86_64. The guest has been resetting over and over most the day, but still no luck.

Comment 14 Radim Krčmář 2014-11-11 21:50:59 UTC
Unlike suberror 1, suberror 3 gives a bit more information

 KVM internal error. Suberror: 3
 extra data[0]: 80000b0e
 extra data[1]: 31

extra data[0] tells that the the event vm-exit occured because of a page fault.
extra data[1] contains EPT misconfiguration exit reason.

EPT misconfiguration is used for MMIO, and we even have an explanation why the internal error occured
  /*
   * Note:
   * Do not try to fix EXIT_REASON_EPT_MISCONFIG if it caused by
   * delivery event since it indicates guest is accessing MMIO.
   * The vm-exit can be triggered again after return to guest that
   * will cause infinite loop.
   */
We might be "accessing MMIO" by accident as we had suberror 1 too ...

RIP+CS is suspicious and you don't see zeroed registers in this context, so a race comes into mind.  Reproducer would help to get an idea.

Comment 17 Radim Krčmář 2016-08-31 01:38:19 UTC
Please preserve the reproducer and reopen this bz if you happen to hit it again.