RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1372581 - CPU feature xsaves is not work properly
Summary: CPU feature xsaves is not work properly
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.3
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Jiri Denemark
QA Contact: chhu
URL:
Whiteboard:
Depends On:
Blocks: libvirtCPUconfig
TreeView+ depends on / blocked
 
Reported: 2016-09-02 06:30 UTC by chhu
Modified: 2017-08-02 01:27 UTC (History)
8 users (show)

Fixed In Version: libvirt-3.2.0-1.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-08-01 17:14:13 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
Qemu log for the rhel7.3 domain (7.26 KB, text/plain)
2016-09-05 02:34 UTC, chhu 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2017:1846 0 normal SHIPPED_LIVE libvirt bug fix and enhancement update 2017-08-01 18:02:50 UTC

Description chhu 2016-09-02 06:30:16 UTC 
Description:
CPU feature xsaves is not work properly.

How reproducible:
100%

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.6.0-22.el7.x86_64
libvirt-2.0.0-6.el7.x86_64

Steps to reproduce:
1. Start a VM with cpu feature: xsaves.
# virsh start rhel7
Domain rhel7 started

# virsh dumpxml rhel7| grep "<cpu" -A 6
  <cpu mode='host-model'>
    <model fallback='allow'/>
    <feature policy='require' name='xsaves'/>
  </cpu>

# ps -ef|grep qemu
qemu      1384     1 29 04:10 ?        00:00:12 /usr/libexec/qemu-kvm -name guest=rhel7...
-cpu Skylake-Client,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,+monitor,+ds_cpl,+vmx,+smx,+est,+tm2,
+xtpr,+pdcm,+osxsave,+tsc_adjust,+clflushopt,+xsaves,+pdpe1gb -m 1024...

2. On host: cat /proc/cpuinfo | grep xsaves, no output, not include xsaves.
but host cpuid shows the XSAVES is enabled, and virsh capabilites, include xsaves.
   # cpuid
      XSAVE features (0xd/1):
      XSAVEOPT instruction                        = true
      XSAVEC instruction                          = true
      XGETBV instruction                          = true
      XSAVES/XRSTORS instructions                 = true
      SAVE area size in bytes                     = 0x000003c0 (960)
      IA32_XSS lower 32 bits valid bit field mask = 0x00000100
      IA32_XSS upper 32 bits valid bit field mask = 0x00000000
  
# virsh capabilities|grep xsaves
      <feature name='xsaves'/>
Login to the guest, cat /proc/cpuinfo, there is no xsaves.

3. Check the feature xsaves is not support migration
# cat /usr/share/libvirt/cpu_map.xml|grep xsaves
    <feature name='xsaves' migratable='no'>

# virsh capabilities >capabilities.xml
# virsh cpu-baseline capabilities.xml | grep xsaves
<feature policy='require' name='xsaves'/>
# virsh cpu-baseline capabilities.xml --migratable| grep xsaves
<no output>

4. Do managedsave and migrate without any error message, the operations should failed with error message, similar as:
"error: Requested operation is not valid: domain has CPU feature xsaves"

1) managedsave: no error message
# virsh managedsave rhel7
Domain rhel7 state saved by libvirt
# virsh start rhel7
Domain rhel7 started
# virsh dumpxml rhel7| grep xsaves
    <feature policy='require' name='xsaves'/>
# ps -ef|grep qemu
qemu      1703     1  0 04:38 ?        00:00:00 /usr/libexec/qemu-kvm -name guest=rhel7...
-cpu Skylake-Client,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,+monitor,+ds_cpl,+vmx,
+smx,+est,+tm2,+xtpr,+pdcm,+osxsave,+tsc_adjust,+clflushopt,+xsaves,+pdpe1gb

2) migration: no error message.
# virsh migrate rhel7 qemu+ssh://ipaddress/system --live --verbose
Migration: [100 %]

 
Actual results:
In step2: the guest start successfully, but the guest cpu not support xsaves.
In step4: do managedsave and migrate, no error message.

Expected results:
In step2: start the guest should fail, or the guest cpu should support xsaves.
In step4: do managedsave and migrate, should failed with error message, as this cpu feature is not migratable.
Comment 2 Jiri Denemark 2016-09-02 08:11:32 UTC 
Is there any difference if you use

<cpu mode='custom' match='exact'>
  <model fallback='forbid'>Skylake-Client</model>
  <feature policy='require' name='xsaves'/>
</cpu>

instead of host-model (which is known to be wrong and buggy in lot of aspects)?
Comment 3 chhu 2016-09-02 10:55:34 UTC 
(In reply to Jiri Denemark from comment #2)
> Is there any difference if you use
> 
> <cpu mode='custom' match='exact'>
>   <model fallback='forbid'>Skylake-Client</model>
>   <feature policy='require' name='xsaves'/>
> </cpu>
> 
> instead of host-model (which is known to be wrong and buggy in lot of
> aspects)?

Thanks Jiri, I tested with the xml above, the test results are the same. Please see details below:

1. Start a VM with cpu feature: xsaves.
# virsh start rhel7.3
Domain rhel7.3 started

# virsh dumpxml rhel7.3|grep "<cpu" -A 6
  <cpu mode='custom' match='exact'>
    <model fallback='forbid'>Skylake-Client</model>
    <feature policy='require' name='xsaves'/>
  </cpu>

# ps -ef|grep qemu
qemu      7190     1 19 06:34 ?        00:00:13 /usr/libexec/qemu-kvm -name guest=rhel7.3,... -cpu Skylake-Client,+xsaves -m 1024...

2. Login to the guest, cat /proc/cpuinfo, there is no xsaves.

3. Do managedsave and migrate without any error message, the operations should failed with error message, similar as:
"error: Requested operation is not valid: domain has CPU feature xsaves"

1) managedsave: no error message
# virsh managedsave rhel7.3
Domain rhel7.3 state saved by libvirt

# virsh start rhel7.3
Domain rhel7.3 started

# virsh dumpxml rhel7.3|grep "<cpu" -A 6
  <cpu mode='custom' match='exact'>
    <model fallback='forbid'>Skylake-Client</model>
    <feature policy='require' name='xsaves'/>
  </cpu>

2) migration: no error message
# virsh dumpxml rhel7.3|grep "<cpu" -A 6
  <cpu mode='custom' match='exact'>
    <model fallback='forbid'>Skylake-Client</model>
    <feature policy='require' name='xsaves'/>
  </cpu>

# virsh migrate rhel7.3 qemu+ssh://ipaddress/system --live --verbose
Migration: [100 %]
Comment 4 Jiri Denemark 2016-09-02 11:21:07 UTC 
Oh and I forgot to ask for qemu log for the rhel7.3 domain. Could you please attach it?
Comment 5 chhu 2016-09-05 02:34:45 UTC 
Created attachment 1197742 [details]
Qemu log for the rhel7.3 domain

Attached the qemu log for the rhel7.3 domain: rhel7.3.log

There is warning message:
"warning: host doesn't support requested feature: CPUID.0DH:EAX.xsaves [bit 3]"

Add more information in above comments here:
1. On host: cat /proc/cpuinfo | grep xsaves, no output, not include xsaves.
but host cpuid shows the XSAVES is enabled, and virsh capabilites, include xsaves.
   # cpuid
      XSAVE features (0xd/1):
      XSAVEOPT instruction                        = true
      XSAVEC instruction                          = true
      XGETBV instruction                          = true
      XSAVES/XRSTORS instructions                 = true
      SAVE area size in bytes                     = 0x000003c0 (960)
      IA32_XSS lower 32 bits valid bit field mask = 0x00000100
      IA32_XSS upper 32 bits valid bit field mask = 0x00000000
  
# virsh capabilities|grep xsaves
      <feature name='xsaves'/>

2. Check the feature xsaves is not support migration
# cat /usr/share/libvirt/cpu_map.xml|grep xsaves
    <feature name='xsaves' migratable='no'>

# virsh capabilities >capabilities.xml
# virsh cpu-baseline capabilities.xml | grep xsaves
<feature policy='require' name='xsaves'/>
# virsh cpu-baseline capabilities.xml --migratable| grep xsaves
<no output>
Comment 6 Jiri Denemark 2016-09-06 10:38:39 UTC 
I see, so the feature is supported by the host CPU, but the kernel/KVM/QEMU combination does not provide support for it. In other words, this is just another reason why we need to fix bug 822148 and bug 824989.
Comment 7 chhu 2016-09-07 05:56:36 UTC 
Add another issue related to non-migratable features:

Imporve the error message:
Migrate with 'invtsc' get qemu error:
"unable to execute QEMU command 'migrate': State blocked by non-migratable device 'cpu'" 

The error message should be reported by libvirt as:
"error: Requested operation is not valid: domain has CPU feature: invtsc"

As the xsaves is also in the qemu-kvm command line, the error message should also include xsaves.

How reproducible:
100%

Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.6.0-22.el7.x86_64
libvirt-2.0.0-6.el7.x86_64
kernel: kernel-3.10.0-495.el7.x86_64

Steps to reproduce:
1. Start a VM with xml below successfully.
# virsh start r7-s1
Domain r7-s1 started

# virsh dumpxml r7-s1|grep "<cpu" -A 6
  <cpu mode='custom' match='minimum'>
    <model fallback='forbid'>Skylake-Client</model>
  </cpu>

# ps -ef|grep r7-s1
qemu     17490     1 11 22:59 ?        00:00:12 /usr/libexec/qemu-kvm -name guest=r7-s1...... -cpu Skylake-Client,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,+monitor,+ds_cpl,+vmx,+smx,+est,+tm2,+xtpr,+pdcm,+osxsave,+tsc_adjust,+clflushopt,+xsaves,+pdpe1gb,+invtsc -m 1024

2. Try to do migration test: met error: "unable to execute QEMU command 'migrate': State blocked by non-migratable device 'cpu'"

# virsh migrate r7-s1 qemu+ssh://10.19.176.166/system --live --verbose
root.176.166's password: 
error: internal error: unable to execute QEMU command 'migrate': State blocked by non-migratable device 'cpu'
Comment 8 chhu 2016-09-13 07:43:36 UTC 
(In reply to chhu from comment #7) 
> Below issue came from bug1365500 comment11.
> Add another issue related to non-migratable features:
> 
> Imporve the error message:
> Migrate with 'invtsc' get qemu error:
> "unable to execute QEMU command 'migrate': State blocked by non-migratable
> device 'cpu'" 
> 
> The error message should be reported by libvirt as:
> "error: Requested operation is not valid: domain has CPU feature: invtsc"
> 
> As the xsaves is also in the qemu-kvm command line, the error message should
> also include xsaves.
> 
> How reproducible:
> 100%
> 
> Version-Release number of selected component (if applicable):
> qemu-kvm-rhev-2.6.0-22.el7.x86_64
> libvirt-2.0.0-6.el7.x86_64
> kernel: kernel-3.10.0-495.el7.x86_64
> 
> Steps to reproduce:
> 1. Start a VM with xml below successfully.
> # virsh start r7-s1
> Domain r7-s1 started
> 
> # virsh dumpxml r7-s1|grep "<cpu" -A 6
>   <cpu mode='custom' match='minimum'>
>     <model fallback='forbid'>Skylake-Client</model>
>   </cpu>
> 
> # ps -ef|grep r7-s1
> qemu     17490     1 11 22:59 ?        00:00:12 /usr/libexec/qemu-kvm -name
> guest=r7-s1...... -cpu
> Skylake-Client,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,+monitor,+ds_cpl,+vmx,+smx,
> +est,+tm2,+xtpr,+pdcm,+osxsave,+tsc_adjust,+clflushopt,+xsaves,+pdpe1gb,
> +invtsc -m 1024
> 
> 2. Try to do migration test: met error: "unable to execute QEMU command
> 'migrate': State blocked by non-migratable device 'cpu'"
> 
> # virsh migrate r7-s1 qemu+ssh://10.19.176.166/system --live --verbose
> root.176.166's password: 
> error: internal error: unable to execute QEMU command 'migrate': State
> blocked by non-migratable device 'cpu'

The same error met while testing on Haswell-noTSX, with xml below:
  <cpu mode='custom' match='minimum'>
    <model fallback='allow'>Haswell-noTSX</model>
  </cpu>
Comment 9 Jiri Denemark 2017-03-03 19:27:53 UTC 
This should be finally fixed by (in combination with QEMU 2.9.0):

commit 2a586b4402a7637e0bef9a2876d065c0ce6bfef1
Refs: v3.1.0-9-g2a586b440
Author:     Jiri Denemark <jdenemar>
AuthorDate: Mon Jan 30 16:10:22 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:56 2017 +0100

    qemucapstest: Update test data for QEMU 2.9.0

    Signed-off-by: Jiri Denemark <jdenemar>

commit 0bde051f3de02b1be25ea4a4d9f062abfa3d1397
Refs: v3.1.0-10-g0bde051f3
Author:     Jiri Denemark <jdenemar>
AuthorDate: Mon Jan 30 16:10:49 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:56 2017 +0100

    domaincapstest: Add test data for QEMU 2.9.0

    Signed-off-by: Jiri Denemark <jdenemar>

commit d2f8f3052d48f284d56e27c98ce7a2ce6c656e59
Refs: v3.1.0-11-gd2f8f3052
Author:     Jiri Denemark <jdenemar>
AuthorDate: Wed Feb 15 10:18:53 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:56 2017 +0100

    docs: Update description of the host-model CPU mode

    Signed-off-by: Jiri Denemark <jdenemar>

commit 4c0723a1d75b981e8939c4c5b6bde7607fc7301e
Refs: v3.1.0-12-g4c0723a1d
Author:     Jiri Denemark <jdenemar>
AuthorDate: Mon Jan 30 16:30:13 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:56 2017 +0100

    qemu: Rename hostCPU/feature element in capabilities cache

    The element will be generalized in the following commits.

    Signed-off-by: Jiri Denemark <jdenemar>

commit 03a34f6b84da009291e8651aba71df8a6761d081
Refs: v3.1.0-13-g03a34f6b8
Author:     Jiri Denemark <jdenemar>
AuthorDate: Wed Feb 22 15:46:47 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:56 2017 +0100

    qemu: Prepare for more types in qemuMonitorCPUModelInfo

    Signed-off-by: Jiri Denemark <jdenemar>

commit 2fc215dd2ad4b88c1054da804c4c45b3d4e5c2fa
Refs: v3.1.0-14-g2fc215dd2
Author:     Jiri Denemark <jdenemar>
AuthorDate: Wed Feb 22 16:01:30 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:56 2017 +0100

    qemu: Store more types in qemuMonitorCPUModelInfo

    While query-cpu-model-expansion returns only boolean features on s390,
    but x86_64 reports some integer and string properties which we are
    interested in.

    Signed-off-by: Jiri Denemark <jdenemar>

commit d7f054a512a911a386d9bbeec51379e4bb843ca5
Refs: v3.1.0-15-gd7f054a51
Author:     Jiri Denemark <jdenemar>
AuthorDate: Wed Feb 22 16:51:50 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:57 2017 +0100

    qemu: Probe "max" CPU model in TCG

    Querying "host" CPU model expansion only makes sense for KVM. QEMU 2.9.0
    introduces a new "max" CPU model which can be used to ask QEMU what the
    best CPU it can provide to a TCG domain is.

    Signed-off-by: Jiri Denemark <jdenemar>

commit f0138289920d5204c1654bc9b17115d1a315d62e
Refs: v3.1.0-16-gf01382899
Author:     Jiri Denemark <jdenemar>
AuthorDate: Wed Jan 11 14:36:34 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:57 2017 +0100

    qemu: Get host CPU model from QEMU on x86_64

    Until now host-model CPU mode tried to enable all CPU features supported
    by the host CPU even if QEMU/KVM did not support them. This caused a
    number of issues and made host-model quite unreliable. Asking QEMU for
    the CPU it can provide and the current host makes host-model much more
    robust.

    This commit fixes the following bugs:

        https://bugzilla.redhat.com/show_bug.cgi?id=1018251
        https://bugzilla.redhat.com/show_bug.cgi?id=1371617
        https://bugzilla.redhat.com/show_bug.cgi?id=1372581
        https://bugzilla.redhat.com/show_bug.cgi?id=1404627
        https://bugzilla.redhat.com/show_bug.cgi?id=870071

    In addition to that, the following bug should be mostly limited to cases
    when an unsupported feature is explicitly requested:

       	https://bugzilla.redhat.com/show_bug.cgi?id=1335534

    Signed-off-by: Jiri Denemark <jdenemar>

commit be3d59754b1a1da174ff1796882a0ceb35e198e8
Refs: v3.1.0-17-gbe3d59754
Author:     Jiri Denemark <jdenemar>
AuthorDate: Tue Jan 31 13:44:00 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:57 2017 +0100

    qemu: Use enum for CPU model expansion type

    Signed-off-by: Jiri Denemark <jdenemar>

commit bb3363c90b5b19c37f8e5b8f512eb00014d2dae4
Refs: v3.1.0-18-gbb3363c90
Author:     Jiri Denemark <jdenemar>
AuthorDate: Thu Feb 23 13:53:51 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:57 2017 +0100

    qemu: Use full CPU model expansion on x86

    The static CPU model expansion is designed to return only canonical
    names of all CPU properties. To maintain backwards compatibility libvirt
    is stuck with different spelling of some of the features, but we need to
    use the full expansion to get the additional spellings. In addition to
    returning all spelling variants for all properties the full expansion
    will contain properties which are not guaranteed to be migration
    compatible. Thus, we need to combine both expansions. First we need to
    call the static expansion to limit the result to migratable properties.
    Then we can use the result of the static expansion as an input to the
    full expansion to get both canonical names and their aliases.

    Signed-off-by: Jiri Denemark <jdenemar>

commit 2f882dbfa92c14d585a786a42d284b63ffdca4e3
Refs: v3.1.0-19-g2f882dbfa
Author:     Jiri Denemark <jdenemar>
AuthorDate: Thu Feb 23 14:31:23 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:57 2017 +0100

    qemu: Make virQEMUCapsInitCPUModel testable

    Signed-off-by: Jiri Denemark <jdenemar>

commit d065934cd07c01fbb29f25bbb223eb4ce126a90e
Refs: v3.1.0-20-gd065934cd
Author:     Jiri Denemark <jdenemar>
AuthorDate: Wed Feb 1 17:48:41 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:57 2017 +0100

    cputest: Switch host CPU data scripts to model expansion

    Instantiating "host" CPU and querying it using qom-get has been the only
    way of probing host CPU via QEMU until 2.9.0 implemented
    query-cpu-model-expansion for x86_64. Even though libvirt never really
    used the old way its result can be easily converted into the one
    produced by query-cpu-model-expansion. Thus we can reuse the original
    test data and possible get new data from hosts where QEMU does not
    support the new QMP command.

    Signed-off-by: Jiri Denemark <jdenemar>

commit d46a1aa4d8caafe977cc41a80ef86af1d10e60b7
Refs: v3.1.0-21-gd46a1aa4d
Author:     Jiri Denemark <jdenemar>
AuthorDate: Mon Feb 13 14:59:42 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:57 2017 +0100

    cputest: Convert all json data files to query-cpu-model-expansion

    Converted by running the following command, renaming the files as
    *.new, and committing only the *.new files.

        (cd tests/cputestdata; ./cpu-convert.py *.json)

    Signed-off-by: Jiri Denemark <jdenemar>

commit a19696b5924e7512dcca4f30d15147036708389e
Refs: v3.1.0-22-ga19696b59
Author:     Jiri Denemark <jdenemar>
AuthorDate: Mon Feb 13 10:33:52 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:57 2017 +0100

    cputest: Test virQEMUCapsInitCPUModel

    The original test didn't use family/model numbers to make better
    decisions about the CPU model and thus mis-detected the model in the two
    cases which are modified in this commit. The detected CPU models now
    match those obtained from raw CPUID data.

    Signed-off-by: Jiri Denemark <jdenemar>

commit 5e4fc2ef993343643587f2b079b63f2c9f038e6f
Refs: v3.1.0-23-g5e4fc2ef9
Author:     Jiri Denemark <jdenemar>
AuthorDate: Mon Feb 13 15:04:38 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:57 2017 +0100

    cputest: Drop obsolete CPU test data files

    Signed-off-by: Jiri Denemark <jdenemar>

commit 8907204cd83f0ca29c48d19bbf2778132d8578a2
Refs: v3.1.0-24-g8907204cd
Author:     Jiri Denemark <jdenemar>
AuthorDate: Mon Feb 13 15:06:35 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:57 2017 +0100

    cputest: Drop .new suffix from CPU test data files

    Signed-off-by: Jiri Denemark <jdenemar>
Comment 11 chhu 2017-06-16 06:54:42 UTC 
Tried to verify on latest libvirt build, but the issues are still existed:

More details:

Tested on packages:
libvirt-3.2.0-10.el7.x86_64
qemu-kvm-rhev-2.9.0-10.el7.x86_64

Steps:
1. Start a VM with cpu featrue: xsaves, the policy will changed to disable in the dumpxml, but the qemu command line shows: xsaves=on, login to the guest, cat /proc/cpuinfo, there is no xsaves.

xml:
  <cpu mode='host-model'>
    <model fallback='allow'/>
    <feature policy='require' name='xsaves'/>
  </cpu>

# virsh start r7-4t
Domain r7-4t started

# virsh dumpxml r7-4t| grep "<cpu" -A 12
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Skylake-Client</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='clflushopt'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='disable' name='xsaves'/>
  </cpu>

# ps -ef|grep qemu| grep r7-4t| grep xsaves
qemu      8353     1  5 01:20 ?        00:00:15 /usr/libexec/qemu-kvm -name guest=r7-4t,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-55-r7-4t/master-key.aes -machine pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off -cpu Skylake-Client,ss=on,hypervisor=on,tsc_adjust=on,clflushopt=on,pdpe1gb=on,xsaves=on ......

2. On host: cat /proc/cpuinfo|grep xsaves, no output, not include xsaves.
   But host cpuid shows the XSAVES is enabled, and virsh capabilites include xsaves.
# cpuid
   ......
   XSAVE features (0xd/1):
      XSAVEOPT instruction                        = true
      XSAVEC instruction                          = true
      XGETBV instruction                          = true
      XSAVES/XRSTORS instructions                 = true
      SAVE area size in bytes                     = 0x000003c0 (960)
      IA32_XSS lower 32 bits valid bit field mask = 0x00000100
      IA32_XSS upper 32 bits valid bit field mask = 0x00000000

# virsh capabilities|grep xsaves
      <feature name='xsaves'/>
Login to the guest, cat /proc/cpuinfo, there is no xsaves.

3. Check the feature xsaves is not support migration
# cat /usr/share/libvirt/cpu_map.xml|grep xsaves
    <feature name='xsaves' migratable='no'>

# virsh capabilities > capabilities.xml
# virsh cpu-baseline capabilities.xml |grep xsaves
  <feature policy='require' name='xsaves'/>
# virsh cpu-baseline capabilities.xml --migratable| grep xsaves
<no output>

4. Do migration without any error message, should failed with error message, as this cpu feature is not migratable.

1) managedsave: after managedsave, start guest, the xsaves=off on the qemu command line.
# virsh managedsave r7-4t
Domain r7-4t state saved by libvirt

# virsh start r7-4t
Domain r7-4t started

# virsh dumpxml r7-4t|grep xsaves
    <feature policy='disable' name='xsaves'/>

# ps -ef|grep qemu| grep r7-4t| grep xsaves
qemu     10127     1  0 01:44 ?        00:00:00 /usr/libexec/qemu-kvm -name guest=r7-4t,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-56-r7-4t/master-key.aes -machine pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off -cpu Skylake-Client,ss=on,hypervisor=on,tsc_adjust=on,clflushopt=on,pdpe1gb=on,xsaves=off.....

Login to the guest, cat /proc/cpuinfo, there is no xsaves.

2)migration: no error message.
# virsh migrate r7-4t qemu+ssh://10.73.73.83/system --live --verbose
Migration: [100 %]

Actual results:
In step2: the guest start with xsaves=on successfully, but the guest cpu not support xsaves.
In step4: do managedsave and migrate, no error message.

Expected results:
In step2: the policy changed to disable in the dumpxml, but the qemu command line shows: xsaves=on
In step4: do migrate, should failed with error message, as this cpu feature is not migratable.


Additional info:
1. Met the same issues when test with cpu mode custom and with xsaves required.

xml:
  <cpu mode='custom' match='exact'>
    <model fallback='forbid'>Skylake-Client</model>
    <feature policy='require' name='xsaves'/>
  </cpu>

# virsh start r7-4t
Domain r7-4t started

# virsh dumpxml r7-4t|grep "<cpu" -A 9
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Skylake-Client</model>
    <feature policy='disable' name='xsaves'/>
    <feature policy='require' name='hypervisor'/>
  </cpu>

# ps -ef|grep r7-4t|grep qemu| grep xsaves
qemu     17384     1 13 02:44 ?        00:00:15 /usr/libexec/qemu-kvm -name guest=r7-4t,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-65-r7-4t/master-key.aes -machine pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off -cpu Skylake-Client,xsaves=on......
Comment 12 chhu 2017-06-16 06:56:49 UTC 
Hi Jiri

I tried to verify on latest libvirt build, but the issues are still existed, details are in comment11, will you please help to have a check? 

Thank you!


Regards,
chhu
Comment 13 chhu 2017-06-16 07:03:19 UTC 
(In reply to chhu from comment #11)
> Tried to verify on latest libvirt build, but the issues are still existed:
> 
> More details:
> 
> Tested on packages:
> libvirt-3.2.0-10.el7.x86_64
> qemu-kvm-rhev-2.9.0-10.el7.x86_64
> 
> Steps:
> 1. Start a VM with cpu featrue: xsaves, the policy will changed to disable
> in the dumpxml, but the qemu command line shows: xsaves=on, login to the
> guest, cat /proc/cpuinfo, there is no xsaves.
> 
> xml:
>   <cpu mode='host-model'>
>     <model fallback='allow'/>
>     <feature policy='require' name='xsaves'/>
>   </cpu>
> 
> # virsh start r7-4t
> Domain r7-4t started
> 
> # virsh dumpxml r7-4t| grep "<cpu" -A 12
>   <cpu mode='custom' match='exact' check='full'>
>     <model fallback='forbid'>Skylake-Client</model>
>     <vendor>Intel</vendor>
>     <feature policy='require' name='ss'/>
>     <feature policy='require' name='hypervisor'/>
>     <feature policy='require' name='tsc_adjust'/>
>     <feature policy='require' name='clflushopt'/>
>     <feature policy='require' name='pdpe1gb'/>
>     <feature policy='disable' name='xsaves'/>
>   </cpu>
> 
> # ps -ef|grep qemu| grep r7-4t| grep xsaves
> qemu      8353     1  5 01:20 ?        00:00:15 /usr/libexec/qemu-kvm -name
> guest=r7-4t,debug-threads=on -S -object
> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-55-r7-4t/
> master-key.aes -machine
> pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off -cpu
> Skylake-Client,ss=on,hypervisor=on,tsc_adjust=on,clflushopt=on,pdpe1gb=on,
> xsaves=on ......
> 
> 2. On host: cat /proc/cpuinfo|grep xsaves, no output, not include xsaves.
>    But host cpuid shows the XSAVES is enabled, and virsh capabilites include
> xsaves.
> # cpuid
>    ......
>    XSAVE features (0xd/1):
>       XSAVEOPT instruction                        = true
>       XSAVEC instruction                          = true
>       XGETBV instruction                          = true
>       XSAVES/XRSTORS instructions                 = true
>       SAVE area size in bytes                     = 0x000003c0 (960)
>       IA32_XSS lower 32 bits valid bit field mask = 0x00000100
>       IA32_XSS upper 32 bits valid bit field mask = 0x00000000
> 
> # virsh capabilities|grep xsaves
>       <feature name='xsaves'/>
> Login to the guest, cat /proc/cpuinfo, there is no xsaves.
> 
> 3. Check the feature xsaves is not support migration
> # cat /usr/share/libvirt/cpu_map.xml|grep xsaves
>     <feature name='xsaves' migratable='no'>
> 
> # virsh capabilities > capabilities.xml
> # virsh cpu-baseline capabilities.xml |grep xsaves
>   <feature policy='require' name='xsaves'/>
> # virsh cpu-baseline capabilities.xml --migratable| grep xsaves
> <no output>
> 
> 4. Do migration without any error message, should failed with error message,
> as this cpu feature is not migratable.
> 
> 1) managedsave: after managedsave, start guest, the xsaves=off on the qemu
> command line.
> # virsh managedsave r7-4t
> Domain r7-4t state saved by libvirt
> 
> # virsh start r7-4t
> Domain r7-4t started
> 
> # virsh dumpxml r7-4t|grep xsaves
>     <feature policy='disable' name='xsaves'/>
> 
> # ps -ef|grep qemu| grep r7-4t| grep xsaves
> qemu     10127     1  0 01:44 ?        00:00:00 /usr/libexec/qemu-kvm -name
> guest=r7-4t,debug-threads=on -S -object
> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-56-r7-4t/
> master-key.aes -machine
> pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off -cpu
> Skylake-Client,ss=on,hypervisor=on,tsc_adjust=on,clflushopt=on,pdpe1gb=on,
> xsaves=off.....
> 
> Login to the guest, cat /proc/cpuinfo, there is no xsaves.
> 
> 2)migration: no error message.
> # virsh migrate r7-4t qemu+ssh://10.73.73.83/system --live --verbose
> Migration: [100 %]
> 
> Actual results:
> In step2: the policy changed to disable in the dumpxml, but the qemu command
> line shows: xsaves=on
> In step4: do managedsave and migrate, no error message.
> 
> Expected results:
> In step2: the policy changed to disable in the dumpxml, the qemu command
> line shows: xsaves=off
> In step4: do migrate, should failed with error message, as this cpu feature
> is not migratable.
> 
> 
> Additional info:
> 1. Met the same issues when test with cpu mode custom and with xsaves
> required.
> 
> xml:
>   <cpu mode='custom' match='exact'>
>     <model fallback='forbid'>Skylake-Client</model>
>     <feature policy='require' name='xsaves'/>
>   </cpu>
> 
> # virsh start r7-4t
> Domain r7-4t started
> 
> # virsh dumpxml r7-4t|grep "<cpu" -A 9
>   <cpu mode='custom' match='exact' check='full'>
>     <model fallback='forbid'>Skylake-Client</model>
>     <feature policy='disable' name='xsaves'/>
>     <feature policy='require' name='hypervisor'/>
>   </cpu>
> 
> # ps -ef|grep r7-4t|grep qemu| grep xsaves
> qemu     17384     1 13 02:44 ?        00:00:15 /usr/libexec/qemu-kvm -name
> guest=r7-4t,debug-threads=on -S -object
> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-65-r7-4t/
> master-key.aes -machine
> pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off -cpu
> Skylake-Client,xsaves=on......
Comment 14 Jiri Denemark 2017-06-16 09:02:20 UTC 
(In reply to chhu from comment #11)
> 1. Start a VM with cpu featrue: xsaves, the policy will changed to disable
> in the dumpxml, but the qemu command line shows: xsaves=on, login to the
> guest, cat /proc/cpuinfo, there is no xsaves.
> 
> xml:
>   <cpu mode='host-model'>
>     <model fallback='allow'/>
>     <feature policy='require' name='xsaves'/>
>   </cpu>

This is asking for a host-model CPU (which can be seen in virsh
domcapabilities) and trying to force xsaves. Since the host CPUID claims
xsaves is present, libvirt won't complain about this. Adding check='full'
would change this, libvirt would refuse the start the domain.

> # virsh start r7-4t
> Domain r7-4t started
> 
> # virsh dumpxml r7-4t| grep "<cpu" -A 12
>   <cpu mode='custom' match='exact' check='full'>
>     <model fallback='forbid'>Skylake-Client</model>
>     <vendor>Intel</vendor>
>     <feature policy='require' name='ss'/>
>     <feature policy='require' name='hypervisor'/>
>     <feature policy='require' name='tsc_adjust'/>
>     <feature policy='require' name='clflushopt'/>
>     <feature policy='require' name='pdpe1gb'/>
>     <feature policy='disable' name='xsaves'/>
>   </cpu>

So this means QEMU/KVM disabled xsave even though the CPUID claims the
features is present and you explicitly requested the feature to be turned on.
QEMU is allowed to do this since you didn't use check='full' when starting the
domain.

> # ps -ef|grep qemu| grep r7-4t| grep xsaves
> qemu      8353     1  5 01:20 ?        00:00:15 /usr/libexec/qemu-kvm -name
> guest=r7-4t,debug-threads=on -S -object
> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-55-r7-4t/
> master-key.aes -machine
> pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off -cpu
> Skylake-Client,ss=on,hypervisor=on,tsc_adjust=on,clflushopt=on,pdpe1gb=on,
> xsaves=on ......

This just reflects the request to enable the feature, it doesn't mean QEMU
will actually enable it. I believe you should see a warning about it in
/var/log/libvirt/qemu/r7-4t.log

> 2. On host: cat /proc/cpuinfo|grep xsaves, no output, not include xsaves.
>    But host cpuid shows the XSAVES is enabled, and virsh capabilites include
> xsaves.
> # cpuid
>    ......
>    XSAVE features (0xd/1):
>       XSAVEOPT instruction                        = true
>       XSAVEC instruction                          = true
>       XGETBV instruction                          = true
>       XSAVES/XRSTORS instructions                 = true
>       SAVE area size in bytes                     = 0x000003c0 (960)
>       IA32_XSS lower 32 bits valid bit field mask = 0x00000100
>       IA32_XSS upper 32 bits valid bit field mask = 0x00000000
> 
> # virsh capabilities|grep xsaves
>       <feature name='xsaves'/>

This is just saying host's CPUID shows xsaves as enabled. You can check virsh
domcapabilities to see if QEMU/KVM is actually able to enable xsaves on this
host or not.

> Login to the guest, cat /proc/cpuinfo, there is no xsaves.

This is correct since QEMU did not enable xsaves for the virtual CPU.

> 3. Check the feature xsaves is not support migration
> # cat /usr/share/libvirt/cpu_map.xml|grep xsaves
>     <feature name='xsaves' migratable='no'>

This is not used with QEMU >= 2.9.0 since we get the migrtability data
directly from QEMU, but it doesn't make any difference here.

> 4. Do migration without any error message, should failed with error message,
> as this cpu feature is not migratable.

No, we detected xsaves is not actually enabled in the virtual CPU so there's
no reason to block the migration.

> 1) managedsave: after managedsave, start guest, the xsaves=off on the qemu
> command line.
> # virsh managedsave r7-4t
> Domain r7-4t state saved by libvirt
> 
> # virsh start r7-4t
> Domain r7-4t started
> 
> # virsh dumpxml r7-4t|grep xsaves
>     <feature policy='disable' name='xsaves'/>
> 
> # ps -ef|grep qemu| grep r7-4t| grep xsaves
> qemu     10127     1  0 01:44 ?        00:00:00 /usr/libexec/qemu-kvm -name
> guest=r7-4t,debug-threads=on -S -object
> secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-56-r7-4t/
> master-key.aes -machine
> pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off -cpu
> Skylake-Client,ss=on,hypervisor=on,tsc_adjust=on,clflushopt=on,pdpe1gb=on,
> xsaves=off.....

This is correct, we already no the xsaves was disabled so we don't enable it
when restoring the domain to maintain guest ABI.

> Login to the guest, cat /proc/cpuinfo, there is no xsaves.

Correct.

> 2)migration: no error message.
> # virsh migrate r7-4t qemu+ssh://10.73.73.83/system --live --verbose
> Migration: [100 %]

Correct.

> Additional info:
> 1. Met the same issues when test with cpu mode custom and with xsaves
> required.
> 
> xml:
>   <cpu mode='custom' match='exact'>
>     <model fallback='forbid'>Skylake-Client</model>
>     <feature policy='require' name='xsaves'/>
>   </cpu>
> 
> # virsh start r7-4t
> Domain r7-4t started
> 
> # virsh dumpxml r7-4t|grep "<cpu" -A 9
>   <cpu mode='custom' match='exact' check='full'>
>     <model fallback='forbid'>Skylake-Client</model>
>     <feature policy='disable' name='xsaves'/>
>     <feature policy='require' name='hypervisor'/>
>   </cpu>

Yeah, this is similar to host-model case. You asked for xsaves, but QEMU did
not enable it.

So far it looks like everything works as expected. Whether QEMU was supposed
to disable xsaves on the host or not is separate question, though. Could you
check that xsaves is really not supported by QEMU/KVM on that host by starting
QEMU directly (without libvirt) with -cpu host,migratable=off and checking
whether the feature is enabled in the guest or not. Or alternatively, remove
all cache files from /var/cache/libvirt/qemu, restart libvirtd with debug
messages enabled and attach the debug logs generated when libvirtd started.
Comment 15 chhu 2017-06-22 08:12:46 UTC 
(In reply to Jiri Denemark from comment #14)
> (In reply to chhu from comment #11)
> > 1. Start a VM with cpu featrue: xsaves, the policy will changed to disable
> > in the dumpxml, but the qemu command line shows: xsaves=on, login to the
> > guest, cat /proc/cpuinfo, there is no xsaves.
> > 
> > xml:
> >   <cpu mode='host-model'>
> >     <model fallback='allow'/>
> >     <feature policy='require' name='xsaves'/>
> >   </cpu>
> 
> This is asking for a host-model CPU (which can be seen in virsh
> domcapabilities) and trying to force xsaves. Since the host CPUID claims
> xsaves is present, libvirt won't complain about this. Adding check='full'
> would change this, libvirt would refuse the start the domain.
> 

The test result is as what your said. Thank you!

Test with packages:
libvirt-3.2.0-14.el7.x86_64
qemu-kvm-rhev-2.9.0-12.el7.x86_64

Test steps:
1. On the host not support xsaves but cpuid shows the XSAVES is enabled, 
try to start a guest with cpu feature:xsaves with check=full, 
libvirt refuse to start the doman: PASS

# virsh dumpxml r7-4t| grep '<cpu' -A 5
  <cpu mode='host-model' check='full'>
    <model fallback='allow'/>
    <feature policy='require' name='xsaves'/>
  </cpu>

# virsh start r7-4t
error: Failed to start domain r7-4t
error: operation failed: guest CPU doesn't match specification: missing features: xsaves

> > # virsh start r7-4t
> > Domain r7-4t started
> > 
> > # virsh dumpxml r7-4t| grep "<cpu" -A 12
> >   <cpu mode='custom' match='exact' check='full'>
> >     <model fallback='forbid'>Skylake-Client</model>
> >     <vendor>Intel</vendor>
> >     <feature policy='require' name='ss'/>
> >     <feature policy='require' name='hypervisor'/>
> >     <feature policy='require' name='tsc_adjust'/>
> >     <feature policy='require' name='clflushopt'/>
> >     <feature policy='require' name='pdpe1gb'/>
> >     <feature policy='disable' name='xsaves'/>
> >   </cpu>
> 
> So this means QEMU/KVM disabled xsave even though the CPUID claims the
> features is present and you explicitly requested the feature to be turned on.
> QEMU is allowed to do this since you didn't use check='full' when starting
> the
> domain.
> 
> > # ps -ef|grep qemu| grep r7-4t| grep xsaves
> > qemu      8353     1  5 01:20 ?        00:00:15 /usr/libexec/qemu-kvm -name
> > guest=r7-4t,debug-threads=on -S -object
> > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-55-r7-4t/
> > master-key.aes -machine
> > pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off -cpu
> > Skylake-Client,ss=on,hypervisor=on,tsc_adjust=on,clflushopt=on,pdpe1gb=on,
> > xsaves=on ......
> 
> This just reflects the request to enable the feature, it doesn't mean QEMU
> will actually enable it. I believe you should see a warning about it in
> /var/log/libvirt/qemu/r7-4t.log

Yes, there is a warning.
-----------------------------------------------------------------------------
On the host not support xsaves but cpuid shows the XSAVES is enabled,
try to start a guest with cpu feature: xsaves without check=full, 
libvirt start the guest, and check in /var/log/libvirt/qemu/r7-4t.log, 
there is a warining:
warning: host doesn't support requested feature: CPUID.0DH:EAX.xsaves [bit 3]
-----------------------------------------------------------------------------

> 
> > 2. On host: cat /proc/cpuinfo|grep xsaves, no output, not include xsaves.
> >    But host cpuid shows the XSAVES is enabled, and virsh capabilites include
> > xsaves.
> > # cpuid
> >    ......
> >    XSAVE features (0xd/1):
> >       XSAVEOPT instruction                        = true
> >       XSAVEC instruction                          = true
> >       XGETBV instruction                          = true
> >       XSAVES/XRSTORS instructions                 = true
> >       SAVE area size in bytes                     = 0x000003c0 (960)
> >       IA32_XSS lower 32 bits valid bit field mask = 0x00000100
> >       IA32_XSS upper 32 bits valid bit field mask = 0x00000000
> > 
> > # virsh capabilities|grep xsaves
> >       <feature name='xsaves'/>
> 
> This is just saying host's CPUID shows xsaves as enabled. You can check virsh
> domcapabilities to see if QEMU/KVM is actually able to enable xsaves on this
> host or not.

Check the virsh domcapabilities, xsaves is not in the list. 

# virsh domcapabilities
  <cpu>
    <mode name='host-passthrough' supported='yes'/>
    <mode name='host-model' supported='yes'>
      <model fallback='forbid'>Skylake-Client</model>
      <vendor>Intel</vendor>
      <feature policy='require' name='ss'/>
      <feature policy='require' name='hypervisor'/>
      <feature policy='require' name='tsc_adjust'/>
      <feature policy='require' name='clflushopt'/>
      <feature policy='require' name='pdpe1gb'/>
      <feature policy='require' name='invtsc'/>
    </mode>
    <mode name='custom' supported='yes'>
            ......
      <model usable='yes'>Skylake-Client</model>
           ......
    </mode>
  </cpu>

> 
> > Login to the guest, cat /proc/cpuinfo, there is no xsaves.
> 
> This is correct since QEMU did not enable xsaves for the virtual CPU.
> 
> > 3. Check the feature xsaves is not support migration
> > # cat /usr/share/libvirt/cpu_map.xml|grep xsaves
> >     <feature name='xsaves' migratable='no'>
> 
> This is not used with QEMU >= 2.9.0 since we get the migrtability data
> directly from QEMU, but it doesn't make any difference here.
> 
> > 4. Do migration without any error message, should failed with error message,
> > as this cpu feature is not migratable.
> 
> No, we detected xsaves is not actually enabled in the virtual CPU so there's
> no reason to block the migration.

Got them, thank you!
Comment 16 chhu 2017-06-22 08:14:00 UTC 
> 
> > 1) managedsave: after managedsave, start guest, the xsaves=off on the qemu
> > command line.
> > # virsh managedsave r7-4t
> > Domain r7-4t state saved by libvirt
> > 
> > # virsh start r7-4t
> > Domain r7-4t started
> > 
> > # virsh dumpxml r7-4t|grep xsaves
> >     <feature policy='disable' name='xsaves'/>
> > 
> > # ps -ef|grep qemu| grep r7-4t| grep xsaves
> > qemu     10127     1  0 01:44 ?        00:00:00 /usr/libexec/qemu-kvm -name
> > guest=r7-4t,debug-threads=on -S -object
> > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-56-r7-4t/
> > master-key.aes -machine
> > pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off -cpu
> > Skylake-Client,ss=on,hypervisor=on,tsc_adjust=on,clflushopt=on,pdpe1gb=on,
> > xsaves=off.....
> 
> This is correct, we already no the xsaves was disabled so we don't enable it
> when restoring the domain to maintain guest ABI.
> 
> > Login to the guest, cat /proc/cpuinfo, there is no xsaves.
> 
> Correct.
> 
> > 2)migration: no error message.
> > # virsh migrate r7-4t qemu+ssh://10.73.73.83/system --live --verbose
> > Migration: [100 %]
> 
> Correct.
> 

Got them, thank you!

> > Additional info:
> > 1. Met the same issues when test with cpu mode custom and with xsaves
> > required.
> > 
> > xml:
> >   <cpu mode='custom' match='exact'>
> >     <model fallback='forbid'>Skylake-Client</model>
> >     <feature policy='require' name='xsaves'/>
> >   </cpu>
> > 
> > # virsh start r7-4t
> > Domain r7-4t started
> > 
> > # virsh dumpxml r7-4t|grep "<cpu" -A 9
> >   <cpu mode='custom' match='exact' check='full'>
> >     <model fallback='forbid'>Skylake-Client</model>
> >     <feature policy='disable' name='xsaves'/>
> >     <feature policy='require' name='hypervisor'/>
> >   </cpu>
> 
> Yeah, this is similar to host-model case. You asked for xsaves, but QEMU did
> not enable it.
> 

Yes, Got the same test results as hot-model case: PASS

1. On the host not support xsaves but cpuid shows the XSAVES is enabled, 
try to start a guest with cpu feature:xsaves with check=full, 
libvirt refuse to start the doman: PASS

# virsh dumpxml r7-4t| grep '<cpu' -A 5
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Skylake-Client</model>
    <feature policy='require' name='xsaves'/>
    <feature policy='require' name='hypervisor'/>
  </cpu>

# virsh start r7-4t
error: Failed to start domain r7-4t
error: operation failed: guest CPU doesn't match specification: missing features: xsaves

2. On the host not support xsaves but cpuid shows the XSAVES is enabled,
try to start a guest with custom mode with cpu feature: xsaves without check=full, libvirt start the guest, and check in /var/log/libvirt/qemu/r7-4t.log, there is a warining:
warning: host doesn't support requested feature: CPUID.0DH:EAX.xsaves [bit 3]

# virsh dumpxml r7-4t| grep '<cpu' -A 5
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Skylake-Client</model>
    <feature policy='disable' name='xsaves'/>
    <feature policy='require' name='hypervisor'/>
  </cpu>
Comment 17 chhu 2017-06-22 09:12:05 UTC 
> So far it looks like everything works as expected. Whether QEMU was supposed
> to disable xsaves on the host or not is separate question, though. Could you
> check that xsaves is really not supported by QEMU/KVM on that host by
> starting
> QEMU directly (without libvirt) with -cpu host,migratable=off and checking
> whether the feature is enabled in the guest or not. Or alternatively, remove
> all cache files from /var/cache/libvirt/qemu, restart libvirtd with debug
> messages enabled and attach the debug logs generated when libvirtd started.

Checked the xsaves is not supported by QEMU/KVM on this host:

Removed all cache files from /var/chache/libvirt/qemu, restart libvirtd with debug messages enabled.

1. Check in the qemu/capabilities file: xsaves with value false.
For x86_64:
-------------------------------------------------------------
<hostCPU type='kvm' model='base' migratability='yes'>
               ......
 381     <property name='xsaves' type='boolean' value='false'/>
-------------------------------------------------------------

More details:
# cat /var/cache/libvirt/qemu/capabilities/3c76bc41d59c0c7314b1ae8e63f4f765d2cf16abaeea081b3ca1f5d8732f7bb1.xml
      1 <qemuCaps>
             ......
    207   <kvmVersion>0</kvmVersion>
    208   <package>(qemu-kvm-rhev-2.9.0-12.el7)</package>
    209   <arch>x86_64</arch>
    210   <hostCPU type='kvm' model='base' migratability='yes'>
    211     <property name='phys-bits' type='number' value='0'/>
                  ......
    381     <property name='xsaves' type='boolean' value='false'/>
                  ......
    437   <hostCPU type='tcg' model='base' migratability='yes'>
                  ......
    608     <property name='xsaves' type='boolean' value='false'/>
    663   </hostCPU>
                   ......
    744 </qemuCaps>

2. Check in libvirtd.log: "xsaves": false

More details:
info : qemuMonitorIOProcess:431 : QEMU_MONITOR_IO_PROCESS: mon=0x7efeac1df740 buf={"return": {"model": {"name": "base", "props": {"cmov": true, "ia64": false, "aes": true, "mmx": true,...... "xsaves": false, "lm": true......

 debug : qemuMonitorJSONIOProcessLine:193 : Line [{"return": {"model": {"name": "base", "props": {"cmov": true, "ia64": false, "aes": true, "mmx": true, "rdpid": false, "arat": true......"xsaves": false, "lm": true,......

debug : virJSONValueFromString:1713 : string={"return": {"model": {"name": "base", "props": {"cmov": true, "ia64": false, "aes": true, "mmx": true, "rdpid": false, "arat": true,......"xsaves": false, "lm": true,......

info : qemuMonitorJSONIOProcessLine:213 : QEMU_MONITOR_RECV_REPLY: mon=0x7efeac1df740 reply={"return": {"model": {"name": "base", "props": {"cmov": true, "ia64": false, "aes": true, "mmx": true,......"xsaves": false, "lm": true,......

 debug : virJSONValueToString:1904 : result={"execute":"query-cpu-model-expansion","arguments":{"type":"full","model":{"name":"base","props":{"cmov":true,"ia64":false,"aes":true,"mmx":true,......"xsaves":false,"lm":true,......

debug : qemuMonitorJSONCommandWithFd:298 : Send command '{"execute":"query-cpu-model-expansion","arguments":{"type":"full","model":{"name":"base","props":{"cmov":true,"ia64":false,"aes":true,"mmx":true,......"xsaves":false,"lm":true,......

 info : qemuMonitorSend:1034 : QEMU_MONITOR_SEND_MSG: mon=0x7efeac1df740 msg={"execute":"query-cpu-model-expansion","arguments":{"type":"full","model":{"name":"base","props":{"cmov":true,"ia64":false,"aes":true,"mmx":true,"rdpid":false,"arat":true,......
"xsaves":false,"lm":true,......

info : qemuMonitorIOWrite:536 : QEMU_MONITOR_IO_WRITE: mon=0x7efeac1df740 buf={"execute":"query-cpu-model-expansion","arguments":{"type":"full","model":{"name":"base","props":{"cmov":true,"ia64":false,"aes":true,"mmx":true,"rdpid":false,"arat":true,......
"xsaves":false,"lm":true,......
Comment 18 chhu 2017-06-22 09:28:59 UTC 
1. (In reply to chhu from comment #7)
> Add another issue related to non-migratable features:
> 
> Imporve the error message:
> Migrate with 'invtsc' get qemu error:
> "unable to execute QEMU command 'migrate': State blocked by non-migratable
> device 'cpu'" 
> 
> The error message should be reported by libvirt as:
> "error: Requested operation is not valid: domain has CPU feature: invtsc"
> 
> As the xsaves is also in the qemu-kvm command line, the error message should
> also include xsaves.
> 
> How reproducible:
> 100%
> 
> Version-Release number of selected component (if applicable):
> qemu-kvm-rhev-2.6.0-22.el7.x86_64
> libvirt-2.0.0-6.el7.x86_64
> kernel: kernel-3.10.0-495.el7.x86_64
> 
> Steps to reproduce:
> 1. Start a VM with xml below successfully.
> # virsh start r7-s1
> Domain r7-s1 started
> 
> # virsh dumpxml r7-s1|grep "<cpu" -A 6
>   <cpu mode='custom' match='minimum'>
>     <model fallback='forbid'>Skylake-Client</model>
>   </cpu>
> 
> # ps -ef|grep r7-s1
> qemu     17490     1 11 22:59 ?        00:00:12 /usr/libexec/qemu-kvm -name
> guest=r7-s1...... -cpu
> Skylake-Client,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,+monitor,+ds_cpl,+vmx,+smx,
> +est,+tm2,+xtpr,+pdcm,+osxsave,+tsc_adjust,+clflushopt,+xsaves,+pdpe1gb,
> +invtsc -m 1024
> 
> 2. Try to do migration test: met error: "unable to execute QEMU command
> 'migrate': State blocked by non-migratable device 'cpu'"
> 
> # virsh migrate r7-s1 qemu+ssh://10.19.176.166/system --live --verbose
> root.176.166's password: 
> error: internal error: unable to execute QEMU command 'migrate': State
> blocked by non-migratable device 'cpu'

Tested on packages:
libvirt-3.2.0-14.el7.x86_64
qemu-kvm-rhev-2.9.0-12.el7.x86_64

Test steps:
1. Define and start a guest with xml below:
  <cpu mode='custom' match='minimum'>
    <model fallback='forbid'>Skylake-Client</model>
  </cpu>

# virsh list --all
 Id    Name                           State
----------------------------------------------------
 45    r7-4t                          running

2. Check the xsaves is not include in qemu command line. As the host not support xsaves: PASS
# virsh dumpxml r7-4t| grep '<cpu' -A 7
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Skylake-Client</model>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='clflushopt'/>
    <feature policy='require' name='pdpe1gb'/>
  </cpu>

# ps -ef|grep r7-4t
qemu     32521     1  2 05:15 ?        00:00:16 /usr/libexec/qemu-kvm -name guest=r7-4t,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-45-r7-4t/master-key.aes -machine pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off -cpu Skylake-Client,ss=on,hypervisor=on,tsc_adjust=on,clflushopt=on,pdpe1gb=on -m 1024 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1......
Comment 19 chhu 2017-06-22 09:43:02 UTC 
(In reply to chhu from comment #18)
> 1. (In reply to chhu from comment #7)
> > Add another issue related to non-migratable features:
> > 
> > Imporve the error message:
> > Migrate with 'invtsc' get qemu error:
> > "unable to execute QEMU command 'migrate': State blocked by non-migratable
> > device 'cpu'" 
> > 
> > The error message should be reported by libvirt as:
> > "error: Requested operation is not valid: domain has CPU feature: invtsc"
> > 

Tested on packages:
libvirt-3.2.0-14.el7.x86_64
qemu-kvm-rhev-2.9.0-12.el7.x86_64

Test steps:
1. Define and start a guest with xml below:
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>Skylake-Client</model>
    <feature policy='require' name='invtsc'/>
    <feature policy='require' name='hypervisor'/>
  </cpu>

# virsh start r7-4t
Domain r7-4t started

2. Try to migrate with invstc, failed with error message: PASS

checked the invtsc is not migratable:
<hostCPU type='kvm' model='base' migratability='yes'>
    <property name='phys-bits' type='number' value='0'/>
      ......
   <property name='invtsc' type='boolean' value='true' migratable='no'/>
      ......

# virsh migrate r7-4t qemu+ssh://10.73.73.83/system --live --verbose
error: Requested operation is not valid: domain has 'invtsc' CPU feature but TSC frequency is not specified
Comment 20 chhu 2017-06-22 09:48:30 UTC 
Thanks Jiri's detailed reply!!

According to the comment 15,16,17,18,19, the bug is verified, set the status to VERIFIED.
Comment 21 errata-xmlrpc 2017-08-01 17:14:13 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846
Comment 22 errata-xmlrpc 2017-08-01 23:55:08 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846
Comment 23 errata-xmlrpc 2017-08-02 01:27:35 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846
Note You need to log in before you can comment on or make changes to this bug.