Bug 1138222

Summary: Fail to managedsave while configure <cpu mode='host-model'> in the guest's xml
Product: Red Hat Enterprise Linux 6 Reporter: vivian zhang <vivianzhang>
Component: libvirtAssignee: Ján Tomko <jtomko>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: medium Docs Contact:
Priority: medium    
Version: 6.6CC: dyuan, jdenemar, jmiao, jsuchane, knoel, libvirt-maint, mail, mtosatti, mzhan, rbalakri, tlavigne, virt-bugs, ydu, zhwang, zpeng
Target Milestone: rcKeywords: Regression
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-0.10.2-46.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: 1138221 Environment:
Last Closed: 2014-10-14 04:23:30 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1138221    
Bug Blocks:    

Description vivian zhang 2014-09-04 10:07:02 UTC
+++ This bug was initially created as a clone of Bug #1138221 +++

Description of problem:
Fail to managedsave while configure <cpu mode='host-model'> in the guest's xml

Version-Release number of selected component (if applicable):
libvirt-1.2.8-1.el7.x86_64
qemu-kvm-rhev-2.1.0-3.el7.x86_64
kernel-3.10.0-150.el7.x86_64


How reproducible:
100%

Steps to Reproduce:

1. prepare a guest with <cpu mode='host-model'>
   #virsh dumpxml rhel7
    --
  <vcpu placement='static'>16</vcpu>
    --
  <cpu mode='host-model'>
    <model fallback='forbid'/>
    <topology sockets='2' cores='4' threads='2'/>
  </cpu>

2. Start the guest
#virsh start rhel7

3. managedsave the guest, will report the following error
   #virsh managedsave rhel7
error: Failed to save domain rhel7 state
error: internal error: unable to execute QEMU command 'migrate': State blocked by non-migratable device 'cpu'

4. it works well on libvirt-1.2.6-1.el7

5.Also hit this issue in rhel6.6

Actual results:
Fail to managedsave while configure <cpu mode='host-model'> in the guest's xml

Expected results:
should succeed to do managedsave

Comment 2 vivian zhang 2014-09-04 10:27:15 UTC
bug also hit on rhel6.6:
libvirt-0.10.2-45.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.441.el6.x86_64
kernel-2.6.32-498.el6.x86_64

bug can not be reproduced on rhel6.5.z: libvirt-0.10.2-29.el6_5.12.x86_64

Comment 3 Jiri Denemark 2014-09-04 13:13:02 UTC
This is a result of invtsc support. When a host CPU supports invtsc, the guest CPU created by host-model will have it too, which results in the QEMU error mentioned in bug description. There is an easy workaround for upstream (adding <feature name='invtsc' policy='disable'/> to guest CPU definition) but it is not supported on RHEL-6. Thus the only workaround in RHEL-6 would be looking at what host-model is translated to (using virsh dumpxml --update-cpu DOMAIN), removing invtsc feature from there and using the result with CPU mode='custom' instead of the original host-model in the domain definition.

Comment 5 Ján Tomko 2014-09-05 12:22:46 UTC
Now pushed upstream:
commit de0aeafe9ce3eb414c8b5d3aa8995d776a2952de
Author:     Ján Tomko <jtomko>
CommitDate: 2014-09-05 12:45:19 +0200

    Don't include non-migratable features in host-model
    
    Commit fba6bc4 introduced support for the 'invtsc' feature,
    which blocks migration. We should not include it in the
    host-model CPU by default, because it's intended to be used
    with migration.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1138221

git describe: v1.2.8-32-gde0aeaf

Comment 7 Marcelo Tosatti 2014-09-05 14:20:20 UTC
(In reply to Jiri Denemark from comment #3)
> This is a result of invtsc support. When a host CPU supports invtsc, the
> guest CPU created by host-model will have it too, which results in the QEMU
> error mentioned in bug description. 

Jiri, this should not be the case: invtsc must be explicitly enabled, as it breaks migration.

IOW, host CPU with invariant TSC support should not expose invariant TSC to a guest, unless explicitly enabled.

Can you confirm this is not the case on RHEL-6.6 ?

> There is an easy workaround for upstream
> (adding <feature name='invtsc' policy='disable'/> to guest CPU definition)
> but it is not supported on RHEL-6. Thus the only workaround in RHEL-6 would
> be looking at what host-model is translated to (using virsh dumpxml
> --update-cpu DOMAIN), removing invtsc feature from there and using the
> result with CPU mode='custom' instead of the original host-model in the
> domain definition.

Comment 8 Jiri Denemark 2014-09-05 14:25:46 UTC
No, "host-model" is not equivalent of -cpu host ("host-passthrough" is its equivalent). This is a bug in libvirt which explicitly enables invtsc for host-model CPUs if a host CPU supports it. That is, we had to teach libvirt not to enable for host-model CPUs because it breaks migration.

Comment 9 Eduardo Habkost 2014-09-05 14:28:26 UTC
(In reply to Marcelo Tosatti from comment #7)
> (In reply to Jiri Denemark from comment #3)
> > This is a result of invtsc support. When a host CPU supports invtsc, the
> > guest CPU created by host-model will have it too, which results in the QEMU
> > error mentioned in bug description. 
> 
> Jiri, this should not be the case: invtsc must be explicitly enabled, as it
> breaks migration.

I don't know if you are talking about libvirt or QEMU. Note that QEMU seems to be doing the right thing, but libvirt was really explicitly enabling invtsc. That's because "host-model" queries the host CPU and generates a -cpu command-line argument enabling all features detected on the host (in other words, this is not about "-cpu host", but something different. "-cpu host" is called "host-passthrough" on libvirt).

Comment 11 zhe peng 2014-09-11 06:53:09 UTC
I can reproduce this with libvirt-0.10.2-45.el6.x86_64
# virsh managedsave rhel6
error: Failed to save domain rhel6 state
error: internal error unable to execute QEMU command 'migrate': An undefined error has ocurred

# virsh migrate rhel6 --live qemu+ssh://10.66.85.72/system --verbose
error: Requested operation is not valid: domain has CPU feature: invtsc

test with build libvirt-0.10.2-46.el6.x86_64

# virsh managedsave rhel6

Domain rhel6 state saved by libvirt

# virsh migrate rhel6 --live qemu+ssh://10.66.85.72/system --verbose
error: unsupported configuration: guest and host CPU are not compatible: Host CPU does not provide required features: rtm, invpcid, erms, bmi2, smep, avx2, hle, bmi1, fsgsbase, abm, pdpe1gb, rdrand, f16c, movbe, fma, monitor

I thought this error because of two host cpu features flags are not same,by now element <feature> not support in host-model. "invtsc" not listed.

so i update xml to:
 <cpu mode='custom' match='exact'>
    <model fallback='allow'>Haswell</model>
    <vendor>Intel</vendor>
    <feature policy='disable' name='rtm'/>
    <feature policy='disable' name='invpcid'/>
    <feature policy='disable' name='erms'/>
    <feature policy='disable' name='bmi2'/>
    <feature policy='disable' name='smep'/>
    <feature policy='disable' name='avx2'/>
    <feature policy='disable' name='hle'/>
    <feature policy='disable' name='bmi1'/>
    <feature policy='disable' name='fsgsbase'/>
    <feature policy='disable' name='abm'/>
    <feature policy='disable' name='pdpe1gb'/>
    <feature policy='disable' name='rdrand'/>
    <feature policy='disable' name='f16c'/>
    <feature policy='disable' name='movbe'/>
    <feature policy='disable' name='fma'/>
    <feature policy='disable' name='monitor'/>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='dtes64'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='xtpr'/>
    <feature policy='require' name='est'/>
    <feature policy='require' name='smx'/>
    <feature policy='require' name='tm'/>
    <feature policy='require' name='acpi'/>
    <feature policy='require' name='osxsave'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='pdcm'/>
    <feature policy='require' name='ds'/>
    <feature policy='require' name='tm2'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='pbe'/>
    <feature policy='require' name='ds_cpl'/>
  </cpu>

# virsh migrate test --live qemu+ssh://10.66.85.72/system --verbose
root.85.72's password: 
Migration: [100 %]

if add <feature policy='require' name='invtsc'/> 
migration will fialed.


Hi Jiri
 Could you help to confirm that verification step is enough? thanks in advance.

Comment 12 Jiri Denemark 2014-09-11 08:03:25 UTC
The guest CPU should be <cpu mode='host-model'/>. The patch for this bug does not do anything for custom mode CPUs.

Comment 13 zhe peng 2014-09-11 08:22:27 UTC
Thanks Jiri's help, my reproduce step and verification used guest xml:
.....  
<cpu mode='host-model'>
    <model fallback='forbid'/>
    <topology sockets='2' cores='4' threads='2'/>
  </cpu>
.....

ignore mode='custom'

managedsave worked well, so move to verified.

Comment 15 errata-xmlrpc 2014-10-14 04:23:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2014-1374.html