Bug 819562

    Technical note added. If any revisions are required, please edit the "Technical Notes" field
    accordingly. All revisions will be proofread by the Engineering Content Services team.
    
    New Contents:
No Documentation Needed

tested bug with qemu-kvm-0.12.1.2-2.290.el6.x86_64

Ivy host(src host) --> SandyBridge host(des host)

steps to reproduce:

1. boot rhel6.3 guest on a host CPU that has the SMEP feature
/usr/libexec/qemu-kvm -M rhel6.3.0 -cpu host --enable-kvm -m 2048M -smp
1,maxcpus=161 -name rhel6.3 -uuid ddcbfb49-3411-1701-3c36-6bdbc00bedbb -rtc
base=utc,clock=host,driftfix=slew -drive
file=/mnt/RHEL-Server-6.3-64-virtio.qcow2,if=none,id=virtio,format=qcow2,cache=none,werror=stop,rerror=stop
-device virtio-blk-pci,drive=virtio,id=drive-virtio0-0-0,bootindex=1 -netdev
tap,id=hostnet1 -device e1000,netdev=hostnet1,id=net1,mac=86:12:50:a4:35:74
-spice port=5912,disable-ticketing -vga qxl -device sga -chardev
socket,id=serial0,path=/var/test3,server,nowait -device
isa-serial,chardev=serial0 -balloon virtio -monitor
unix:/tmp/monitor3,server,nowait -monitor stdio

2. check guest if support SMEP
   guest support SMEP

3. boot guest in des host that doesn't have the SMEP feature

4. do migrate 

Actual results:
migration successful, but guest show internal-error in des host via monitor and
can not resume via "continue" 

(qemu) info status
VM status: paused (internal-error) 



Hi,Eduardo

Did I reproduce this issue?

BTW, seem I hit the following issue.
Bug 816787 - internal-error on dest host after migration finished at getting
pxe config file list

Fu, I think we can ack this bug if we can reproduce that guest can get SMEP when boot it with -M rhel6.2.0 on 6.3 IvyBridge host(after Bug 719818 is fixed).

FuXiangChun: yes, you reproduced one of the scenarios.

Let me extend the expected results list from comment #0:

Expected results:
A) Migration shouldn't be possible if the SMEP feature is enabled and the
destination host doesn't support it.
B) -Mrhel6.2.0 guests must not have the SMEP feature.
C) SMEP feature must not be enabled unless explicitly asked for, or "-cpu host" is used. Even -Mrhel6.3.0 guests can't have it automatically enabled otherwise migration to non-IvyBridge hosts will fail.

(In reply to comment #4)
> FuXiangChun: yes, you reproduced one of the scenarios.
> 
> Let me extend the expected results list from comment #0:
> 
> Expected results:
> A) Migration shouldn't be possible if the SMEP feature is enabled and the
> destination host doesn't support it.
> B) -Mrhel6.2.0 guests must not have the SMEP feature.
reproduced
> C) SMEP feature must not be enabled unless explicitly asked for, or "-cpu host"
We tried default cpu model, smep can not be enabled in guest.
> is used. Even -Mrhel6.3.0 guests can't have it automatically enabled otherwise
> migration to non-IvyBridge hosts will fail.

Ok, we will test all of these three scenarios when we verify this issue.

(In reply to comment #4)
> FuXiangChun: yes, you reproduced one of the scenarios.
> 
> Let me extend the expected results list from comment #0:
> 
> Expected results:
> A) Migration shouldn't be possible if the SMEP feature is enabled and the
> destination host doesn't support it.

Hi, Eduardo,
migration with -cpu host from high host to low host is not supported, do we need to add this kind of scenerios to make sure migration failed?

> B) -Mrhel6.2.0 guests must not have the SMEP feature.

> C) SMEP feature must not be enabled unless explicitly asked for,

--> seems -cpu model_name,+smep is not supported currently.

> or "-cpu host"
> is used. Even -Mrhel6.3.0 guests can't have it automatically enabled otherwise
> migration to non-IvyBridge hosts will fail.

comment2 is tested with -cpu host, so this scenerios should be tested with "-M rhel6.3.0 -cpu Sandybridge" on Ivybridge host to check if smep is enabled?

Marking as regression, as it breaks existing guests if they run on Ivy Bridge hosts (even if older QEMU CPU models are used and the user don't ask for SMEP explicitly).

Fix submitted upstream:
http://marc.info/?l=qemu-devel&m=133720204301452&w=2

New patch submitted upstream:
http://article.gmane.org/gmane.comp.emulators.kvm.devel/91040

v3 submitted upstream: http://marc.info/?l=kvm&m=133761058122125&w=2

tested this bug with qemu-kvm-0.12.1.2-2.294.el6.x86_64 

A) Migration shouldn't be possible if the SMEP feature is enabled and the
destination host doesn't support it

src host show:
(qemu) info migrate
Migration status: completed

des host show:
from monitor in des host
(qemu) infokvm: unhandled exit 80000021
kvm_run returned -22
kvm: unhandled exit 80000021
kvm_run returned -22
kvm: unhandled exit 80000021
kvm_run returned -22
kvm: unhandled exit 80000021
kvm_run returned -22
and guest hang in des host.


Is it expected result?

B) -Mrhel6.2.0 guests must not have the SMEP feature.
  boot guest with "-M rhel6.2.0 -cpu host"
testing result:
 fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes xsave avx f16c rdrand hypervisor lahf_lm xsaveopt fsgsbase smep

smep can be exposed to guest.

C) SMEP feature must not be enabled unless explicitly asked for, or "-cpu host" is used. Even -Mrhel6.3.0 guests can't have it automatically enabled otherwise migration to non-IvyBridge hosts will fail

 There are four scenarios SMEP can be enabled.
1)-M rhel6.3.0 -cpu SandyBridge
guest:
fpu de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc arch_perfmon unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes xsave avx hypervisor lahf_lm xsaveopt fsgsbase smep

guest
eax in: 0x00000007, eax = 00000000 ebx = 00000281 ecx = 00000000 edx = 00000000


2) -M rhel6.3.0 -cpu host
guest:
 fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc arch_perfmon unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes xsave avx f16c rdrand hypervisor lahf_lm xsaveopt fsgsbase smep

guest
eax in: 0x00000007, eax = 00000000 ebx = 00000281 ecx = 00000000 edx = 00000000

3) -M rhel6.2.0 -cpu host
fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes xsave avx f16c rdrand hypervisor lahf_lm xsaveopt fsgsbase smep

guest
eax in: 0x00000007, eax = 00000000 ebx = 00000281 ecx = 00000000 edx = 00000000

4)-M rhel6.3.0 -cpu Westmere
fpu de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx lm constant_tsc arch_perfmon unfair_spinlock pni ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes hypervisor lahf_lm fsgsbase smep

guest
eax in: 0x00000007, eax = 00000000 ebx = 00000281 ecx = 00000000 edx = 00000000

Testing result are different from scenario B and C,according to comment 4 expected results. this bug seem isn't fixed.

(In reply to comment #18)
> tested this bug with qemu-kvm-0.12.1.2-2.294.el6.x86_64 

Note that this is a version without the fix.

> 
> A) Migration shouldn't be possible if the SMEP feature is enabled and the
> destination host doesn't support it
> 
> src host show:
> (qemu) info migrate
> Migration status: completed
> 
> des host show:
> from monitor in des host
> (qemu) infokvm: unhandled exit 80000021
> kvm_run returned -22
> kvm: unhandled exit 80000021
> kvm_run returned -22
> kvm: unhandled exit 80000021
> kvm_run returned -22
> kvm: unhandled exit 80000021
> kvm_run returned -22
> and guest hang in des host.
> 
> 
> Is it expected result?

Yes, and no. If SMEP is enabled, migration being prevented is the expected result, but SMEP simply shouldn't be enabled unless explicitly asked for, so you wouldn't even be able to reproduce the migration failure above anymore, after the fix (except if using "-cpu host", but migration isn't expected to work with "-cpu host", anyway).

> 
> B) -Mrhel6.2.0 guests must not have the SMEP feature.
>   boot guest with "-M rhel6.2.0 -cpu host"
> testing result:
>  fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush
> mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc unfair_spinlock pni
> pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes xsave avx f16c rdrand
> hypervisor lahf_lm xsaveopt fsgsbase smep
> 
> smep can be exposed to guest.

Yes, this is a bug that has to be fixed.


> 
> C) SMEP feature must not be enabled unless explicitly asked for, or "-cpu
> host" is used. Even -Mrhel6.3.0 guests can't have it automatically enabled
> otherwise migration to non-IvyBridge hosts will fail
> 
>  There are four scenarios SMEP can be enabled.
> 1)-M rhel6.3.0 -cpu SandyBridge
> guest:
> fpu de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush mmx
> fxsr sse sse2 ht syscall nx lm constant_tsc arch_perfmon unfair_spinlock pni
> pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes xsave avx hypervisor
> lahf_lm xsaveopt fsgsbase smep
> 
> guest
> eax in: 0x00000007, eax = 00000000 ebx = 00000281 ecx = 00000000 edx =
> 00000000

This is also a bug, and should be fixed on qemu-kvm-0.12.1.2-2.295.el6.

> 
> 
> 2) -M rhel6.3.0 -cpu host
> guest:
>  fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush
> mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc arch_perfmon
> unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes
> xsave avx f16c rdrand hypervisor lahf_lm xsaveopt fsgsbase smep
> 
> guest
> eax in: 0x00000007, eax = 00000000 ebx = 00000281 ecx = 00000000 edx =
> 00000000

This is expected.

> 
> 3) -M rhel6.2.0 -cpu host
> fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush
> mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc unfair_spinlock pni
> pclmulqdq ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes xsave avx f16c rdrand
> hypervisor lahf_lm xsaveopt fsgsbase smep
> 
> guest
> eax in: 0x00000007, eax = 00000000 ebx = 00000281 ecx = 00000000 edx =
> 00000000

This is expected.


> 
> 4)-M rhel6.3.0 -cpu Westmere
> fpu de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush mmx
> fxsr sse sse2 ht syscall nx lm constant_tsc arch_perfmon unfair_spinlock pni
> ssse3 cx16 sse4_1 sse4_2 x2apic popcnt aes hypervisor lahf_lm fsgsbase smep
> 
> guest
> eax in: 0x00000007, eax = 00000000 ebx = 00000281 ecx = 00000000 edx =
> 00000000

This is also a bug, and should be fixed on qemu-kvm-0.12.1.2-2.295.el6.

> 
> Testing result are different from scenario B and C,according to comment 4
> expected results. this bug seem isn't fixed.

It really isn't fixed on qemu-kvm-0.12.1.2-2.294.el6. Please test qemu-kvm-0.12.1.2-2.295.el6.

re-tested this issue with qemu-kvm-0.12.1.2-2.295.el6.x86_64 

below are testing scenarios:
1)migrate
src host show:
(qemu) info migrate
Migration status: completed

des host show:
 from monitor in des host
 (qemu) infokvm: unhandled exit 80000021
 kvm_run returned -22
 kvm: unhandled exit 80000021
 kvm_run returned -22
 kvm: unhandled exit 80000021
 kvm_run returned -22
 kvm: unhandled exit 80000021
 kvm_run returned -22
 and guest hang in des host.

2)-M rhel6.3.0 -cpu SandyBridge
smep isn't exposed to guest
eax in: 0x00000007, eax = 00000000 ebx = 00000000 ecx = 00000000 edx = 00000000

3) -M rhel6.3.0 -cpu host
smep is exposed to guest
eax in: 0x00000007, eax = 00000000 ebx = 00000281 ecx = 00000000 edx =00000000

4) -M rhel6.2.0 -cpu host
smep is exposed to guest
eax in: 0x00000007, eax = 00000000 ebx = 00000281 ecx = 00000000 edx =00000000

5)-M rhel6.3.0 -cpu Westmere

smep isn't exposed to guest
eax in: 0x00000007, eax = 00000000 ebx = 00000000 ecx = 00000000 edx = 00000000

base on above testing result, I think this bug is fixed.

(In reply to comment #21)
> re-tested this issue with qemu-kvm-0.12.1.2-2.295.el6.x86_64 
> 
> below are testing scenarios:
> 1)migrate
> src host show:
> (qemu) info migrate
> Migration status: completed
> 
> des host show:
>  from monitor in des host
>  (qemu) infokvm: unhandled exit 80000021
>  kvm_run returned -22
>  kvm: unhandled exit 80000021
>  kvm_run returned -22
>  kvm: unhandled exit 80000021
>  kvm_run returned -22
>  kvm: unhandled exit 80000021
>  kvm_run returned -22
>  and guest hang in des host.
> 

The above must not happen unless you are using "-cpu host". Were you using "-cpu host" on the test above?

Ivy Bridge => Sandy Bridge migration (and the reverse way too) must work normally with all Intel CPU models defined on QEMU, except "-cpu host".

> 4) -M rhel6.2.0 -cpu host
> smep is exposed to guest
> eax in: 0x00000007, eax = 00000000 ebx = 00000281 ecx = 00000000 edx
> =00000000
> 
----snip logs from commment0---
Expected results:
B) -Mrhel6.2.0 guests must not have the SMEP feature.

According to this is comment, seems should not be exposed to guest when -M rhel6.2.0 -cpu host, however, according to my testing, smep is exposed to guest 
eax in: 0x00000007, eax = 00000000 ebx = 00000281 ecx = 00000000 edx
00000000
Hi, Eduardo

would you please tell me why?

> The above must not happen unless you are using "-cpu host". Were you using
> "-cpu host" on the test above?

Yes, both destination and source with "-cpu host"
> 
> Ivy Bridge => Sandy Bridge migration (and the reverse way too) must work
ok, I will quick testing by using -cpu Sandy bridge, will update results later.
> normally with all Intel CPU models defined on QEMU, except "-cpu host".
we do not support migration by using -cpu host? if yes, why? or we have a existed bug?

Best Regards & Thanks,
Junyi

(In reply to comment #24)
> we do not support migration by using -cpu host? if yes, why? or we have a
> existed bug?

It's by design. -cpu host enables every single feature supported by the host hardware+kernel, even if it QEMU doesn't know what they mean or what guest-visible state is involved. Because of that, QEMU can't make sure migration is safe if using -cpu host.

(In reply to comment #23)
> > 4) -M rhel6.2.0 -cpu host
> > smep is exposed to guest
> > eax in: 0x00000007, eax = 00000000 ebx = 00000281 ecx = 00000000 edx
> > =00000000
> > 
> ----snip logs from commment0---
> Expected results:
> B) -Mrhel6.2.0 guests must not have the SMEP feature.
> 
> According to this is comment, seems should not be exposed to guest when -M
> rhel6.2.0 -cpu host, however, according to my testing, smep is exposed to
> guest 
> eax in: 0x00000007, eax = 00000000 ebx = 00000281 ecx = 00000000 edx
> 00000000

"-cpu host" is an exception: it does not depend on machine-type, because it is not expected to result in exactly the same virtual machine when using the same command-line on different hosts. So this is OK and expected.

(This is an example of why we can't support migration using "-cpu host": running "-cpu host" on Ivy Bridge will expose SMEP, running "-cpu host" on Sandy Bridge won't expose SMEP).

> Ivy Bridge => Sandy Bridge migration (and the reverse way too) must work
> normally with all Intel CPU models defined on QEMU, except "-cpu host".

Tested Ivy Bridge=>Sandy Bridge migration with "-cpu SandyBridge", migration is successful,and guest work well in des host.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2012-0746.html