Bug 1059496

Summary: KVM: x86 emulator: Implement jmp far opcode ff/5
Product: Red Hat Enterprise Linux 6 Reporter: Laszlo Ersek <lersek>
Component: kernelAssignee: Laszlo Ersek <lersek>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.5CC: juzhang, lersek, mtosatti, pbonzini, qzhang
Target Milestone: rc   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: kernel-2.6.32-450.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-10-14 05:53:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
Screenshot for comment 8, case 3. none

Description Laszlo Ersek 2014-01-30 00:06:40 UTC 
*** Description of problem:

Recent upstream versions of qemu come with IPXE option ROMs (pc-bios/efi-*.rom) that contain instructions like

 3e6:   ff 2e                   ljmp   *(%esi)

This instruction is executed in the guest (as part of IPXE) when SeaBIOS decides to boot the NIC's option ROM (which is IPXE).

The instruction is executed without problems (== natively) on physical RHEL-6 hosts that support the "unrestricted_guest" option (== modparam of kvm-intel). 

On older hosts that lack this hardware feature, or disable the "unrestricted_guest" option, it is necessary to pass "emulate_invalid_guest_state=1" to kvm-intel, in order to substitute the native execution with emulation. Without this latter parameter, KVM rejects the emulation of invalid guest state, with hardware exit reason EXIT_REASON_INVALID_STATE (80000021).

Once "emulate_invalid_guest_state=1" is passed, the bug appears: the guest makes no progress, and the following kernel message is logged:

  emulation failed (emulation failure) rip 225 ff 2e 4e 00

See the original report in the thread

  http://lists.nongnu.org/archive/html/qemu-devel/2014-01/msg03921.html

*** Version-Release number of selected component (if applicable):

2.6.32-431.3.1.el6.x86_64 (most recently released RHEL-6.5.z)

*** How reproducible:

100%

*** Steps to Reproduce:

(1) install upstream qemu:
(1a) cd /tmp
(1b) git clone git://git.qemu.org/qemu.git
(1c) cd qemu
(1d) git checkout 0169c511554cb0014a00290b0d3d26c31a49818f
(1e) ./configure --target-list=x86_64-softmmu --prefix=/tmp/qemu-inst
(1f) make -j4
(1g) make install
(1h) cp -av pc-bios/bios-256k.bin /tmp/qemu-inst/share/qemu/

(2) make sure invalid guest state is emulated, not natively executed,
    regardless of host CPU features:
(2a) rmmod kvm_intel kvm
(2b) modprobe kvm_intel unrestricted_guest=0 emulate_invalid_guest_state=1

(3) execute IPXE in the guest:
    /tmp/qemu-inst/bin/qemu-system-x86_64 -enable-kvm -m 1024 \
        -device e1000,netdev=netdev0 -netdev user,id=netdev0

*** Actual results:

After step (3), in the guest (QEMU) window, IPXE hangs (spins) after printing the following message:

  iPXE (PCI 00:03.0) starting execution...

and the dmesg contains

  emulation failed (emulation failure) rip 225 ff 2e 4e 00

*** Expected results:

After step (3), in the guest (QEMU) window, IPXE and SeaBIOS print:

  iPXE (PCI 00:03.0) starting execution...ok
  iPXE initialising devices...ok
  [...]
  DHCP (net0 52:54:00:12:34:45)...... ok
  [...]
  Nothing to boot: [...]
  No more network devices
  No bootable device.

The guest appears hung at this point, but this is intended.

RHEL-6 should support development and testing of upstream qemu.

*** Additional info:

This is fixed by backporting

  <http://thread.gmane.org/gmane.comp.emulators.kvm.devel/47489>

Patch 1:

  commit e35b7b9c9e7d8768ee34e5904fed4cb0f2c2cb5d
  Author: Gleb Natapov <gleb>
  Date:   Thu Feb 25 16:36:42 2010 +0200

      KVM: x86 emulator: Add decoding of 16bit second in memory argument

      Add decoding of Ep type of argument used by callf/jmpf.

      Signed-off-by: Gleb Natapov <gleb>
      Signed-off-by: Avi Kivity <avi>

Patch 2:

  commit ea79849d4c8461034b75acb19c8041b6fddee2a5
  Author: Gleb Natapov <gleb>
  Date:   Thu Feb 25 16:36:43 2010 +0200

      KVM: x86 emulator: Implement jmp far opcode ff/5

      Implement jmp far opcode ff/5. It is used by multiboot loader.

      Signed-off-by: Gleb Natapov <gleb>
      Signed-off-by: Avi Kivity <avi>
Comment 2 Qunfang Zhang 2014-02-21 10:08:34 UTC 
Reproduced on 2.6.32-431.7.1.el6.x86_64. 

Steps:

1. [root@localhost qzhang]# cat /sys/module/kvm_intel/parameters/emulate_invalid_guest_state
N
[root@localhost qzhang]# cat /sys/module/kvm_intel/parameters/unrestricted_guest 
Y
[root@localhost qzhang]# lsmod | grep kvm_intel
kvm_intel              54285  4 
kvm                   332980  1 kvm_intel
[root@localhost qzhang]# 
[root@localhost qzhang]# rmmod kvm_intel
[root@localhost qzhang]# modprobe kvm_intel unrestricted_guest=0 emulate_invalid_guest_state=1
[root@localhost qzhang]#  cat /sys/module/kvm_intel/parameters/emulate_invalid_guest_state
Y
[root@localhost qzhang]#  cat /sys/module/kvm_intel/parameters/unrestricted_guest 
N
[root@localhost qzhang]# /usr/libexec/qemu-kvm  -enable-kvm -m 1024 -device e1000,netdev=netdev0 -netdev user,id=netdev0
VNC server running on `::1:5900'

Result: 
Guest hangs and displays the following message:
  iPXE (PCI 00:03.0) starting execution...

Host dmesg: 
emulation failed (emulation failure) rip 2e5 ff 2e 7e 00
emulation failed (emulation failure) rip 2e5 ff 2e 7e 00
emulation failed (emulation failure) rip 2e5 ff 2e 7e 00
emulation failed (emulation failure) rip 2e5 ff 2e 7e 00
emulation failed (emulation failure) rip 2e5 ff 2e 7e 00
emulation failed (emulation failure) rip 2e5 ff 2e 7e 00
emulation failed (emulation failure) rip 2e5 ff 2e 7e 00
emulation failed (emulation failure) rip 2e5 ff 2e 7e 00
emulation failed (emulation failure) rip 2e5 ff 2e 7e 00
emulation failed (emulation failure) rip 2e5 ff 2e 7e 00
......
Comment 3 Rafael Aquini 2014-03-19 00:10:23 UTC 
Patch(es) available on kernel-2.6.32-450.el6
Comment 6 Qunfang Zhang 2014-06-25 10:48:05 UTC 
Reproduced on 2.6.32-431.21.1.el6.x86_64. 

Steps:

1. [root@localhost]# cat /sys/module/kvm_intel/parameters/emulate_invalid_guest_state
N
[root@localhost]# cat /sys/module/kvm_intel/parameters/unrestricted_guest 
Y
[root@localhost]# lsmod | grep kvm_intel
kvm_intel              54285  4 
kvm                   332980  1 kvm_intel
[root@localhost]# 
[root@localhost]# rmmod kvm_intel
[root@localhost]# modprobe kvm_intel unrestricted_guest=0 emulate_invalid_guest_state=1
[root@localhost]#  cat /sys/module/kvm_intel/parameters/emulate_invalid_guest_state
Y
[root@localhost]#  cat /sys/module/kvm_intel/parameters/unrestricted_guest 
N

2. Start a command line: 

[root@localhost]# /usr/libexec/qemu-kvm  -enable-kvm -m 1024 -device e1000,netdev=netdev0 -netdev user,id=netdev0
VNC server running on `::1:5900'

Result: 
(1) Qemu-kvm hangs.
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                    
 2301 root      20   0 1286m  25m 4988 S 100.2  0.3   0:23.77 qemu-kvm                                                                  
    1 root      20   0 19356 1536 1228 S  0.0  0.0   0:00.69 init   

(2) Host dmesg:
[root@localhost ~]# dmesg 
emulation failed (emulation failure) rip 2e5 ff 2e 7e 00
emulation failed (emulation failure) rip 2e5 ff 2e 7e 00
emulation failed (emulation failure) rip 2e5 ff 2e 7e 00
emulation failed (emulation failure) rip 2e5 ff 2e 7e 00
emulation failed (emulation failure) rip 2e5 ff 2e 7e 00
emulation failed (emulation failure) rip 2e5 ff 2e 7e 00
emulation failed (emulation failure) rip 2e5 ff 2e 7e 00
emulation failed (emulation failure) rip 2e5 ff 2e 7e 00
emulation failed (emulation failure) rip 2e5 ff 2e 7e 00
emulation failed (emulation failure) rip 2e5 ff 2e 7e 00


Verified on kernel-2.6.32-487.el6.x86_64 with the same steps as above.

Result:
After step 2, after start the command line, the qemu-kvm does not hang. And host dmesg does not show up the above message. 


Hi, Laszlo

Is this enough to call the bug verified pass? 

Thanks!
Qunfang
Comment 7 Laszlo Ersek 2014-06-25 11:06:12 UTC 
Qunfang,

please also check that iPXE actually progresses beyond the "starting execution" message:

(In reply to Laszlo Ersek from comment #0)
> *** Expected results:
> 
> After step (3), in the guest (QEMU) window, IPXE and SeaBIOS print:
> 
>   iPXE (PCI 00:03.0) starting execution...ok
>   iPXE initialising devices...ok
>   [...]
>   DHCP (net0 52:54:00:12:34:45)...... ok
>   [...]
>   Nothing to boot: [...]
>   No more network devices
>   No bootable device.
> 
> The guest appears hung at this point, but this is intended.

Thanks.
Laszlo
Comment 8 Qunfang Zhang 2014-06-26 08:07:59 UTC 
Hi, Laszlo

I re-test again with same steps in comment 6.

1. Reproduced on kernel-2.6.32-431.21.1.el6.x86_64:

Result:  Qemu-kvm hangs at "iPXE (PCI 00:03.0) starting execution..." and qemu consumes 100% cpu. Host dmesg prompts the serial of the following log "emulation failed (emulation failure) rip 8d24 0f 00 d0 b0"

CLI:

[root@localhost]# /usr/libexec/qemu-kvm  -enable-kvm -m 1024 -device e1000,netdev=netdev0 -netdev user,id=netdev0

2. Test the new version kernel-2.6.32-487.el6.x86_64:

CLI:
[root@localhost]# /usr/libexec/qemu-kvm  -enable-kvm -m 1024 -device e1000,netdev=netdev0 -netdev user,id=netdev0

Result: Qemu-kvm does not hang at the point of "iPXE (PCI 00:03.0) starting execution...", and the dmesg does not output the serial of "emulation failed (emulation failure) rip 8d24 0f 00 d0 b0" message. 


3. Still test the kernel-2.6.32-487.el6.x86_64:

CLI: (boot from network)

usr/libexec/qemu-kvm -cpu SandyBridge -M rhel6.5.0 -enable-kvm -m 4096 -smp 4,sockets=2,cores=2,threads=1 -name rhel6.4-64 -uuid 9a0e67ec-f286-d8e7-0548-0c1c9ec93009 -nodefconfig -nodefaults  -rtc base=utc,clock=host,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 -drive file=/home/disk.qcow2,if=none,id=drive-virtio-disk0,format=qcow2,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,id=hostnet0,vhost=on -device virtio-net-pci,netdev=hostnet0,id=net0,mac=e2:64:7e:60:dd:6d,bus=pci.0,addr=0x3,bootindex=1 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=channel1,path=/tmp/helloworld1,server,nowait -device virtserialport,chardev=channel1,name=port1,bus=virtio-serial0.0,id=port1 -chardev socket,id=channel2,path=/tmp/helloworld2,server,nowait -device virtserialport,chardev=channel2,name=port2,bus=virtio-serial0.0,id=port2 -device usb-tablet,id=input0 -vnc :10 -vga std -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6  -drive if=none,id=drive-fdc0-0-0,format=raw,cache=none -global isa-fdc.driveA=drive-fdc0-0-0 -qmp tcp:0:5555,server,nowait -global PIIX4_PM.disable_s3=0 -global PIIX4_PM.disable_s4=0 -monitor stdio

Result:

(1) Host dmesg:

emulation failed (emulation failure) rip 8d24 0f 00 d0 b0
emulation failed (emulation failure) rip 8d24 0f 00 d0 b0
emulation failed (emulation failure) rip 8d24 0f 00 d0 b0
......

(2) qemu-kvm consumes 100% cpu

(3) Guest desktop hangs before displaying the pxe profile list. (please refer to the attachment). 

Laszlo,

So, the case 2 seems to be a verified pass result, however for the case 3 (just with a different command line), there's still problem. Right? 

Thanks,
Qunfang
Comment 9 Qunfang Zhang 2014-06-26 08:08:55 UTC 
Created attachment 912369 [details]
Screenshot for comment 8, case 3.
Comment 10 Laszlo Ersek 2014-06-26 08:45:43 UTC 
Qunfang,

the

  Out of memory parsing config file

message (visible on the screenshot in comment 9) comes from SYSLINUX, not iPXE.

(Looking at the SYSLINUX repository, this message was introduced in commit 67af2ad7, in the year 2004. Later, the message was removed in commit 14531c47, which was released in "syslinux-5.00". I just checked Brew, and the latest build packaged for RHEL-6 is "syslinux-4.04-3.el6". This means that the RHEL-6 version of SYSLINUX still contains the message.)

What does it mean that the message comes from SYSLINUX, not iPXE? It means that iPXE has booted SYSLINUX successfully, and only SYSLINUX fails for some reason. (In addition, the emulation failure triggered by SYSLINUX, "rip 8d24 0f 00 d0 b0", is different from "rip 225 ff 2e 4e 00" in comment 0 -- ie. it is a different instruction from what the BZ is about.)

Hence the latest results seem to confirm that the original issue reported in comment 0 has been fixed. The emulation problem in SYSLINUX is out of scope for this bug. (And fixing *all* possible emulation problems is likely out of scope for RHEL-6 entirely...) Thanks.
Comment 11 Qunfang Zhang 2014-06-26 09:57:12 UTC 
Hi, Laszlo

Thanks a lot for the explanation. So I will verify this bug according to comment 6, comment 8 and comment 10.

Thanks.
Comment 12 errata-xmlrpc 2014-10-14 05:53:56 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHSA-2014-1392.html