Bug 1372581
| Summary: | CPU feature xsaves is not work properly | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | chhu | ||||
| Component: | libvirt | Assignee: | Jiri Denemark <jdenemar> | ||||
| Status: | CLOSED ERRATA | QA Contact: | chhu | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 7.3 | CC: | chhu, dyuan, jdenemar, jpriddy, lhuang, rbalakri, xuzhang, yalzhang | ||||
| Target Milestone: | rc | Keywords: | Upstream | ||||
| Target Release: | --- | ||||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | libvirt-3.2.0-1.el7 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2017-08-01 17:14:13 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1199452 | ||||||
| Attachments: |
|
||||||
Is there any difference if you use <cpu mode='custom' match='exact'> <model fallback='forbid'>Skylake-Client</model> <feature policy='require' name='xsaves'/> </cpu> instead of host-model (which is known to be wrong and buggy in lot of aspects)? (In reply to Jiri Denemark from comment #2) > Is there any difference if you use > > <cpu mode='custom' match='exact'> > <model fallback='forbid'>Skylake-Client</model> > <feature policy='require' name='xsaves'/> > </cpu> > > instead of host-model (which is known to be wrong and buggy in lot of > aspects)? Thanks Jiri, I tested with the xml above, the test results are the same. Please see details below: 1. Start a VM with cpu feature: xsaves. # virsh start rhel7.3 Domain rhel7.3 started # virsh dumpxml rhel7.3|grep "<cpu" -A 6 <cpu mode='custom' match='exact'> <model fallback='forbid'>Skylake-Client</model> <feature policy='require' name='xsaves'/> </cpu> # ps -ef|grep qemu qemu 7190 1 19 06:34 ? 00:00:13 /usr/libexec/qemu-kvm -name guest=rhel7.3,... -cpu Skylake-Client,+xsaves -m 1024... 2. Login to the guest, cat /proc/cpuinfo, there is no xsaves. 3. Do managedsave and migrate without any error message, the operations should failed with error message, similar as: "error: Requested operation is not valid: domain has CPU feature xsaves" 1) managedsave: no error message # virsh managedsave rhel7.3 Domain rhel7.3 state saved by libvirt # virsh start rhel7.3 Domain rhel7.3 started # virsh dumpxml rhel7.3|grep "<cpu" -A 6 <cpu mode='custom' match='exact'> <model fallback='forbid'>Skylake-Client</model> <feature policy='require' name='xsaves'/> </cpu> 2) migration: no error message # virsh dumpxml rhel7.3|grep "<cpu" -A 6 <cpu mode='custom' match='exact'> <model fallback='forbid'>Skylake-Client</model> <feature policy='require' name='xsaves'/> </cpu> # virsh migrate rhel7.3 qemu+ssh://ipaddress/system --live --verbose Migration: [100 %] Oh and I forgot to ask for qemu log for the rhel7.3 domain. Could you please attach it? Created attachment 1197742 [details]
Qemu log for the rhel7.3 domain
Attached the qemu log for the rhel7.3 domain: rhel7.3.log
There is warning message:
"warning: host doesn't support requested feature: CPUID.0DH:EAX.xsaves [bit 3]"
Add more information in above comments here:
1. On host: cat /proc/cpuinfo | grep xsaves, no output, not include xsaves.
but host cpuid shows the XSAVES is enabled, and virsh capabilites, include xsaves.
# cpuid
XSAVE features (0xd/1):
XSAVEOPT instruction = true
XSAVEC instruction = true
XGETBV instruction = true
XSAVES/XRSTORS instructions = true
SAVE area size in bytes = 0x000003c0 (960)
IA32_XSS lower 32 bits valid bit field mask = 0x00000100
IA32_XSS upper 32 bits valid bit field mask = 0x00000000
# virsh capabilities|grep xsaves
<feature name='xsaves'/>
2. Check the feature xsaves is not support migration
# cat /usr/share/libvirt/cpu_map.xml|grep xsaves
<feature name='xsaves' migratable='no'>
# virsh capabilities >capabilities.xml
# virsh cpu-baseline capabilities.xml | grep xsaves
<feature policy='require' name='xsaves'/>
# virsh cpu-baseline capabilities.xml --migratable| grep xsaves
<no output>
I see, so the feature is supported by the host CPU, but the kernel/KVM/QEMU combination does not provide support for it. In other words, this is just another reason why we need to fix bug 822148 and bug 824989. Add another issue related to non-migratable features:
Imporve the error message:
Migrate with 'invtsc' get qemu error:
"unable to execute QEMU command 'migrate': State blocked by non-migratable device 'cpu'"
The error message should be reported by libvirt as:
"error: Requested operation is not valid: domain has CPU feature: invtsc"
As the xsaves is also in the qemu-kvm command line, the error message should also include xsaves.
How reproducible:
100%
Version-Release number of selected component (if applicable):
qemu-kvm-rhev-2.6.0-22.el7.x86_64
libvirt-2.0.0-6.el7.x86_64
kernel: kernel-3.10.0-495.el7.x86_64
Steps to reproduce:
1. Start a VM with xml below successfully.
# virsh start r7-s1
Domain r7-s1 started
# virsh dumpxml r7-s1|grep "<cpu" -A 6
<cpu mode='custom' match='minimum'>
<model fallback='forbid'>Skylake-Client</model>
</cpu>
# ps -ef|grep r7-s1
qemu 17490 1 11 22:59 ? 00:00:12 /usr/libexec/qemu-kvm -name guest=r7-s1...... -cpu Skylake-Client,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,+monitor,+ds_cpl,+vmx,+smx,+est,+tm2,+xtpr,+pdcm,+osxsave,+tsc_adjust,+clflushopt,+xsaves,+pdpe1gb,+invtsc -m 1024
2. Try to do migration test: met error: "unable to execute QEMU command 'migrate': State blocked by non-migratable device 'cpu'"
# virsh migrate r7-s1 qemu+ssh://10.19.176.166/system --live --verbose
root.176.166's password:
error: internal error: unable to execute QEMU command 'migrate': State blocked by non-migratable device 'cpu'
(In reply to chhu from comment #7) > Below issue came from bug1365500 comment11. > Add another issue related to non-migratable features: > > Imporve the error message: > Migrate with 'invtsc' get qemu error: > "unable to execute QEMU command 'migrate': State blocked by non-migratable > device 'cpu'" > > The error message should be reported by libvirt as: > "error: Requested operation is not valid: domain has CPU feature: invtsc" > > As the xsaves is also in the qemu-kvm command line, the error message should > also include xsaves. > > How reproducible: > 100% > > Version-Release number of selected component (if applicable): > qemu-kvm-rhev-2.6.0-22.el7.x86_64 > libvirt-2.0.0-6.el7.x86_64 > kernel: kernel-3.10.0-495.el7.x86_64 > > Steps to reproduce: > 1. Start a VM with xml below successfully. > # virsh start r7-s1 > Domain r7-s1 started > > # virsh dumpxml r7-s1|grep "<cpu" -A 6 > <cpu mode='custom' match='minimum'> > <model fallback='forbid'>Skylake-Client</model> > </cpu> > > # ps -ef|grep r7-s1 > qemu 17490 1 11 22:59 ? 00:00:12 /usr/libexec/qemu-kvm -name > guest=r7-s1...... -cpu > Skylake-Client,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,+monitor,+ds_cpl,+vmx,+smx, > +est,+tm2,+xtpr,+pdcm,+osxsave,+tsc_adjust,+clflushopt,+xsaves,+pdpe1gb, > +invtsc -m 1024 > > 2. Try to do migration test: met error: "unable to execute QEMU command > 'migrate': State blocked by non-migratable device 'cpu'" > > # virsh migrate r7-s1 qemu+ssh://10.19.176.166/system --live --verbose > root.176.166's password: > error: internal error: unable to execute QEMU command 'migrate': State > blocked by non-migratable device 'cpu' The same error met while testing on Haswell-noTSX, with xml below: <cpu mode='custom' match='minimum'> <model fallback='allow'>Haswell-noTSX</model> </cpu> This should be finally fixed by (in combination with QEMU 2.9.0):
commit 2a586b4402a7637e0bef9a2876d065c0ce6bfef1
Refs: v3.1.0-9-g2a586b440
Author: Jiri Denemark <jdenemar>
AuthorDate: Mon Jan 30 16:10:22 2017 +0100
Commit: Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:56 2017 +0100
qemucapstest: Update test data for QEMU 2.9.0
Signed-off-by: Jiri Denemark <jdenemar>
commit 0bde051f3de02b1be25ea4a4d9f062abfa3d1397
Refs: v3.1.0-10-g0bde051f3
Author: Jiri Denemark <jdenemar>
AuthorDate: Mon Jan 30 16:10:49 2017 +0100
Commit: Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:56 2017 +0100
domaincapstest: Add test data for QEMU 2.9.0
Signed-off-by: Jiri Denemark <jdenemar>
commit d2f8f3052d48f284d56e27c98ce7a2ce6c656e59
Refs: v3.1.0-11-gd2f8f3052
Author: Jiri Denemark <jdenemar>
AuthorDate: Wed Feb 15 10:18:53 2017 +0100
Commit: Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:56 2017 +0100
docs: Update description of the host-model CPU mode
Signed-off-by: Jiri Denemark <jdenemar>
commit 4c0723a1d75b981e8939c4c5b6bde7607fc7301e
Refs: v3.1.0-12-g4c0723a1d
Author: Jiri Denemark <jdenemar>
AuthorDate: Mon Jan 30 16:30:13 2017 +0100
Commit: Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:56 2017 +0100
qemu: Rename hostCPU/feature element in capabilities cache
The element will be generalized in the following commits.
Signed-off-by: Jiri Denemark <jdenemar>
commit 03a34f6b84da009291e8651aba71df8a6761d081
Refs: v3.1.0-13-g03a34f6b8
Author: Jiri Denemark <jdenemar>
AuthorDate: Wed Feb 22 15:46:47 2017 +0100
Commit: Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:56 2017 +0100
qemu: Prepare for more types in qemuMonitorCPUModelInfo
Signed-off-by: Jiri Denemark <jdenemar>
commit 2fc215dd2ad4b88c1054da804c4c45b3d4e5c2fa
Refs: v3.1.0-14-g2fc215dd2
Author: Jiri Denemark <jdenemar>
AuthorDate: Wed Feb 22 16:01:30 2017 +0100
Commit: Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:56 2017 +0100
qemu: Store more types in qemuMonitorCPUModelInfo
While query-cpu-model-expansion returns only boolean features on s390,
but x86_64 reports some integer and string properties which we are
interested in.
Signed-off-by: Jiri Denemark <jdenemar>
commit d7f054a512a911a386d9bbeec51379e4bb843ca5
Refs: v3.1.0-15-gd7f054a51
Author: Jiri Denemark <jdenemar>
AuthorDate: Wed Feb 22 16:51:50 2017 +0100
Commit: Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:57 2017 +0100
qemu: Probe "max" CPU model in TCG
Querying "host" CPU model expansion only makes sense for KVM. QEMU 2.9.0
introduces a new "max" CPU model which can be used to ask QEMU what the
best CPU it can provide to a TCG domain is.
Signed-off-by: Jiri Denemark <jdenemar>
commit f0138289920d5204c1654bc9b17115d1a315d62e
Refs: v3.1.0-16-gf01382899
Author: Jiri Denemark <jdenemar>
AuthorDate: Wed Jan 11 14:36:34 2017 +0100
Commit: Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:57 2017 +0100
qemu: Get host CPU model from QEMU on x86_64
Until now host-model CPU mode tried to enable all CPU features supported
by the host CPU even if QEMU/KVM did not support them. This caused a
number of issues and made host-model quite unreliable. Asking QEMU for
the CPU it can provide and the current host makes host-model much more
robust.
This commit fixes the following bugs:
https://bugzilla.redhat.com/show_bug.cgi?id=1018251
https://bugzilla.redhat.com/show_bug.cgi?id=1371617
https://bugzilla.redhat.com/show_bug.cgi?id=1372581
https://bugzilla.redhat.com/show_bug.cgi?id=1404627
https://bugzilla.redhat.com/show_bug.cgi?id=870071
In addition to that, the following bug should be mostly limited to cases
when an unsupported feature is explicitly requested:
https://bugzilla.redhat.com/show_bug.cgi?id=1335534
Signed-off-by: Jiri Denemark <jdenemar>
commit be3d59754b1a1da174ff1796882a0ceb35e198e8
Refs: v3.1.0-17-gbe3d59754
Author: Jiri Denemark <jdenemar>
AuthorDate: Tue Jan 31 13:44:00 2017 +0100
Commit: Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:57 2017 +0100
qemu: Use enum for CPU model expansion type
Signed-off-by: Jiri Denemark <jdenemar>
commit bb3363c90b5b19c37f8e5b8f512eb00014d2dae4
Refs: v3.1.0-18-gbb3363c90
Author: Jiri Denemark <jdenemar>
AuthorDate: Thu Feb 23 13:53:51 2017 +0100
Commit: Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:57 2017 +0100
qemu: Use full CPU model expansion on x86
The static CPU model expansion is designed to return only canonical
names of all CPU properties. To maintain backwards compatibility libvirt
is stuck with different spelling of some of the features, but we need to
use the full expansion to get the additional spellings. In addition to
returning all spelling variants for all properties the full expansion
will contain properties which are not guaranteed to be migration
compatible. Thus, we need to combine both expansions. First we need to
call the static expansion to limit the result to migratable properties.
Then we can use the result of the static expansion as an input to the
full expansion to get both canonical names and their aliases.
Signed-off-by: Jiri Denemark <jdenemar>
commit 2f882dbfa92c14d585a786a42d284b63ffdca4e3
Refs: v3.1.0-19-g2f882dbfa
Author: Jiri Denemark <jdenemar>
AuthorDate: Thu Feb 23 14:31:23 2017 +0100
Commit: Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:57 2017 +0100
qemu: Make virQEMUCapsInitCPUModel testable
Signed-off-by: Jiri Denemark <jdenemar>
commit d065934cd07c01fbb29f25bbb223eb4ce126a90e
Refs: v3.1.0-20-gd065934cd
Author: Jiri Denemark <jdenemar>
AuthorDate: Wed Feb 1 17:48:41 2017 +0100
Commit: Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:57 2017 +0100
cputest: Switch host CPU data scripts to model expansion
Instantiating "host" CPU and querying it using qom-get has been the only
way of probing host CPU via QEMU until 2.9.0 implemented
query-cpu-model-expansion for x86_64. Even though libvirt never really
used the old way its result can be easily converted into the one
produced by query-cpu-model-expansion. Thus we can reuse the original
test data and possible get new data from hosts where QEMU does not
support the new QMP command.
Signed-off-by: Jiri Denemark <jdenemar>
commit d46a1aa4d8caafe977cc41a80ef86af1d10e60b7
Refs: v3.1.0-21-gd46a1aa4d
Author: Jiri Denemark <jdenemar>
AuthorDate: Mon Feb 13 14:59:42 2017 +0100
Commit: Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:57 2017 +0100
cputest: Convert all json data files to query-cpu-model-expansion
Converted by running the following command, renaming the files as
*.new, and committing only the *.new files.
(cd tests/cputestdata; ./cpu-convert.py *.json)
Signed-off-by: Jiri Denemark <jdenemar>
commit a19696b5924e7512dcca4f30d15147036708389e
Refs: v3.1.0-22-ga19696b59
Author: Jiri Denemark <jdenemar>
AuthorDate: Mon Feb 13 10:33:52 2017 +0100
Commit: Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:57 2017 +0100
cputest: Test virQEMUCapsInitCPUModel
The original test didn't use family/model numbers to make better
decisions about the CPU model and thus mis-detected the model in the two
cases which are modified in this commit. The detected CPU models now
match those obtained from raw CPUID data.
Signed-off-by: Jiri Denemark <jdenemar>
commit 5e4fc2ef993343643587f2b079b63f2c9f038e6f
Refs: v3.1.0-23-g5e4fc2ef9
Author: Jiri Denemark <jdenemar>
AuthorDate: Mon Feb 13 15:04:38 2017 +0100
Commit: Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:57 2017 +0100
cputest: Drop obsolete CPU test data files
Signed-off-by: Jiri Denemark <jdenemar>
commit 8907204cd83f0ca29c48d19bbf2778132d8578a2
Refs: v3.1.0-24-g8907204cd
Author: Jiri Denemark <jdenemar>
AuthorDate: Mon Feb 13 15:06:35 2017 +0100
Commit: Jiri Denemark <jdenemar>
CommitDate: Fri Mar 3 19:57:57 2017 +0100
cputest: Drop .new suffix from CPU test data files
Signed-off-by: Jiri Denemark <jdenemar>
Tried to verify on latest libvirt build, but the issues are still existed:
More details:
Tested on packages:
libvirt-3.2.0-10.el7.x86_64
qemu-kvm-rhev-2.9.0-10.el7.x86_64
Steps:
1. Start a VM with cpu featrue: xsaves, the policy will changed to disable in the dumpxml, but the qemu command line shows: xsaves=on, login to the guest, cat /proc/cpuinfo, there is no xsaves.
xml:
<cpu mode='host-model'>
<model fallback='allow'/>
<feature policy='require' name='xsaves'/>
</cpu>
# virsh start r7-4t
Domain r7-4t started
# virsh dumpxml r7-4t| grep "<cpu" -A 12
<cpu mode='custom' match='exact' check='full'>
<model fallback='forbid'>Skylake-Client</model>
<vendor>Intel</vendor>
<feature policy='require' name='ss'/>
<feature policy='require' name='hypervisor'/>
<feature policy='require' name='tsc_adjust'/>
<feature policy='require' name='clflushopt'/>
<feature policy='require' name='pdpe1gb'/>
<feature policy='disable' name='xsaves'/>
</cpu>
# ps -ef|grep qemu| grep r7-4t| grep xsaves
qemu 8353 1 5 01:20 ? 00:00:15 /usr/libexec/qemu-kvm -name guest=r7-4t,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-55-r7-4t/master-key.aes -machine pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off -cpu Skylake-Client,ss=on,hypervisor=on,tsc_adjust=on,clflushopt=on,pdpe1gb=on,xsaves=on ......
2. On host: cat /proc/cpuinfo|grep xsaves, no output, not include xsaves.
But host cpuid shows the XSAVES is enabled, and virsh capabilites include xsaves.
# cpuid
......
XSAVE features (0xd/1):
XSAVEOPT instruction = true
XSAVEC instruction = true
XGETBV instruction = true
XSAVES/XRSTORS instructions = true
SAVE area size in bytes = 0x000003c0 (960)
IA32_XSS lower 32 bits valid bit field mask = 0x00000100
IA32_XSS upper 32 bits valid bit field mask = 0x00000000
# virsh capabilities|grep xsaves
<feature name='xsaves'/>
Login to the guest, cat /proc/cpuinfo, there is no xsaves.
3. Check the feature xsaves is not support migration
# cat /usr/share/libvirt/cpu_map.xml|grep xsaves
<feature name='xsaves' migratable='no'>
# virsh capabilities > capabilities.xml
# virsh cpu-baseline capabilities.xml |grep xsaves
<feature policy='require' name='xsaves'/>
# virsh cpu-baseline capabilities.xml --migratable| grep xsaves
<no output>
4. Do migration without any error message, should failed with error message, as this cpu feature is not migratable.
1) managedsave: after managedsave, start guest, the xsaves=off on the qemu command line.
# virsh managedsave r7-4t
Domain r7-4t state saved by libvirt
# virsh start r7-4t
Domain r7-4t started
# virsh dumpxml r7-4t|grep xsaves
<feature policy='disable' name='xsaves'/>
# ps -ef|grep qemu| grep r7-4t| grep xsaves
qemu 10127 1 0 01:44 ? 00:00:00 /usr/libexec/qemu-kvm -name guest=r7-4t,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-56-r7-4t/master-key.aes -machine pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off -cpu Skylake-Client,ss=on,hypervisor=on,tsc_adjust=on,clflushopt=on,pdpe1gb=on,xsaves=off.....
Login to the guest, cat /proc/cpuinfo, there is no xsaves.
2)migration: no error message.
# virsh migrate r7-4t qemu+ssh://10.73.73.83/system --live --verbose
Migration: [100 %]
Actual results:
In step2: the guest start with xsaves=on successfully, but the guest cpu not support xsaves.
In step4: do managedsave and migrate, no error message.
Expected results:
In step2: the policy changed to disable in the dumpxml, but the qemu command line shows: xsaves=on
In step4: do migrate, should failed with error message, as this cpu feature is not migratable.
Additional info:
1. Met the same issues when test with cpu mode custom and with xsaves required.
xml:
<cpu mode='custom' match='exact'>
<model fallback='forbid'>Skylake-Client</model>
<feature policy='require' name='xsaves'/>
</cpu>
# virsh start r7-4t
Domain r7-4t started
# virsh dumpxml r7-4t|grep "<cpu" -A 9
<cpu mode='custom' match='exact' check='full'>
<model fallback='forbid'>Skylake-Client</model>
<feature policy='disable' name='xsaves'/>
<feature policy='require' name='hypervisor'/>
</cpu>
# ps -ef|grep r7-4t|grep qemu| grep xsaves
qemu 17384 1 13 02:44 ? 00:00:15 /usr/libexec/qemu-kvm -name guest=r7-4t,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-65-r7-4t/master-key.aes -machine pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off -cpu Skylake-Client,xsaves=on......
Hi Jiri I tried to verify on latest libvirt build, but the issues are still existed, details are in comment11, will you please help to have a check? Thank you! Regards, chhu (In reply to chhu from comment #11) > Tried to verify on latest libvirt build, but the issues are still existed: > > More details: > > Tested on packages: > libvirt-3.2.0-10.el7.x86_64 > qemu-kvm-rhev-2.9.0-10.el7.x86_64 > > Steps: > 1. Start a VM with cpu featrue: xsaves, the policy will changed to disable > in the dumpxml, but the qemu command line shows: xsaves=on, login to the > guest, cat /proc/cpuinfo, there is no xsaves. > > xml: > <cpu mode='host-model'> > <model fallback='allow'/> > <feature policy='require' name='xsaves'/> > </cpu> > > # virsh start r7-4t > Domain r7-4t started > > # virsh dumpxml r7-4t| grep "<cpu" -A 12 > <cpu mode='custom' match='exact' check='full'> > <model fallback='forbid'>Skylake-Client</model> > <vendor>Intel</vendor> > <feature policy='require' name='ss'/> > <feature policy='require' name='hypervisor'/> > <feature policy='require' name='tsc_adjust'/> > <feature policy='require' name='clflushopt'/> > <feature policy='require' name='pdpe1gb'/> > <feature policy='disable' name='xsaves'/> > </cpu> > > # ps -ef|grep qemu| grep r7-4t| grep xsaves > qemu 8353 1 5 01:20 ? 00:00:15 /usr/libexec/qemu-kvm -name > guest=r7-4t,debug-threads=on -S -object > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-55-r7-4t/ > master-key.aes -machine > pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off -cpu > Skylake-Client,ss=on,hypervisor=on,tsc_adjust=on,clflushopt=on,pdpe1gb=on, > xsaves=on ...... > > 2. On host: cat /proc/cpuinfo|grep xsaves, no output, not include xsaves. > But host cpuid shows the XSAVES is enabled, and virsh capabilites include > xsaves. > # cpuid > ...... > XSAVE features (0xd/1): > XSAVEOPT instruction = true > XSAVEC instruction = true > XGETBV instruction = true > XSAVES/XRSTORS instructions = true > SAVE area size in bytes = 0x000003c0 (960) > IA32_XSS lower 32 bits valid bit field mask = 0x00000100 > IA32_XSS upper 32 bits valid bit field mask = 0x00000000 > > # virsh capabilities|grep xsaves > <feature name='xsaves'/> > Login to the guest, cat /proc/cpuinfo, there is no xsaves. > > 3. Check the feature xsaves is not support migration > # cat /usr/share/libvirt/cpu_map.xml|grep xsaves > <feature name='xsaves' migratable='no'> > > # virsh capabilities > capabilities.xml > # virsh cpu-baseline capabilities.xml |grep xsaves > <feature policy='require' name='xsaves'/> > # virsh cpu-baseline capabilities.xml --migratable| grep xsaves > <no output> > > 4. Do migration without any error message, should failed with error message, > as this cpu feature is not migratable. > > 1) managedsave: after managedsave, start guest, the xsaves=off on the qemu > command line. > # virsh managedsave r7-4t > Domain r7-4t state saved by libvirt > > # virsh start r7-4t > Domain r7-4t started > > # virsh dumpxml r7-4t|grep xsaves > <feature policy='disable' name='xsaves'/> > > # ps -ef|grep qemu| grep r7-4t| grep xsaves > qemu 10127 1 0 01:44 ? 00:00:00 /usr/libexec/qemu-kvm -name > guest=r7-4t,debug-threads=on -S -object > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-56-r7-4t/ > master-key.aes -machine > pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off -cpu > Skylake-Client,ss=on,hypervisor=on,tsc_adjust=on,clflushopt=on,pdpe1gb=on, > xsaves=off..... > > Login to the guest, cat /proc/cpuinfo, there is no xsaves. > > 2)migration: no error message. > # virsh migrate r7-4t qemu+ssh://10.73.73.83/system --live --verbose > Migration: [100 %] > > Actual results: > In step2: the policy changed to disable in the dumpxml, but the qemu command > line shows: xsaves=on > In step4: do managedsave and migrate, no error message. > > Expected results: > In step2: the policy changed to disable in the dumpxml, the qemu command > line shows: xsaves=off > In step4: do migrate, should failed with error message, as this cpu feature > is not migratable. > > > Additional info: > 1. Met the same issues when test with cpu mode custom and with xsaves > required. > > xml: > <cpu mode='custom' match='exact'> > <model fallback='forbid'>Skylake-Client</model> > <feature policy='require' name='xsaves'/> > </cpu> > > # virsh start r7-4t > Domain r7-4t started > > # virsh dumpxml r7-4t|grep "<cpu" -A 9 > <cpu mode='custom' match='exact' check='full'> > <model fallback='forbid'>Skylake-Client</model> > <feature policy='disable' name='xsaves'/> > <feature policy='require' name='hypervisor'/> > </cpu> > > # ps -ef|grep r7-4t|grep qemu| grep xsaves > qemu 17384 1 13 02:44 ? 00:00:15 /usr/libexec/qemu-kvm -name > guest=r7-4t,debug-threads=on -S -object > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-65-r7-4t/ > master-key.aes -machine > pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off -cpu > Skylake-Client,xsaves=on...... (In reply to chhu from comment #11) > 1. Start a VM with cpu featrue: xsaves, the policy will changed to disable > in the dumpxml, but the qemu command line shows: xsaves=on, login to the > guest, cat /proc/cpuinfo, there is no xsaves. > > xml: > <cpu mode='host-model'> > <model fallback='allow'/> > <feature policy='require' name='xsaves'/> > </cpu> This is asking for a host-model CPU (which can be seen in virsh domcapabilities) and trying to force xsaves. Since the host CPUID claims xsaves is present, libvirt won't complain about this. Adding check='full' would change this, libvirt would refuse the start the domain. > # virsh start r7-4t > Domain r7-4t started > > # virsh dumpxml r7-4t| grep "<cpu" -A 12 > <cpu mode='custom' match='exact' check='full'> > <model fallback='forbid'>Skylake-Client</model> > <vendor>Intel</vendor> > <feature policy='require' name='ss'/> > <feature policy='require' name='hypervisor'/> > <feature policy='require' name='tsc_adjust'/> > <feature policy='require' name='clflushopt'/> > <feature policy='require' name='pdpe1gb'/> > <feature policy='disable' name='xsaves'/> > </cpu> So this means QEMU/KVM disabled xsave even though the CPUID claims the features is present and you explicitly requested the feature to be turned on. QEMU is allowed to do this since you didn't use check='full' when starting the domain. > # ps -ef|grep qemu| grep r7-4t| grep xsaves > qemu 8353 1 5 01:20 ? 00:00:15 /usr/libexec/qemu-kvm -name > guest=r7-4t,debug-threads=on -S -object > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-55-r7-4t/ > master-key.aes -machine > pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off -cpu > Skylake-Client,ss=on,hypervisor=on,tsc_adjust=on,clflushopt=on,pdpe1gb=on, > xsaves=on ...... This just reflects the request to enable the feature, it doesn't mean QEMU will actually enable it. I believe you should see a warning about it in /var/log/libvirt/qemu/r7-4t.log > 2. On host: cat /proc/cpuinfo|grep xsaves, no output, not include xsaves. > But host cpuid shows the XSAVES is enabled, and virsh capabilites include > xsaves. > # cpuid > ...... > XSAVE features (0xd/1): > XSAVEOPT instruction = true > XSAVEC instruction = true > XGETBV instruction = true > XSAVES/XRSTORS instructions = true > SAVE area size in bytes = 0x000003c0 (960) > IA32_XSS lower 32 bits valid bit field mask = 0x00000100 > IA32_XSS upper 32 bits valid bit field mask = 0x00000000 > > # virsh capabilities|grep xsaves > <feature name='xsaves'/> This is just saying host's CPUID shows xsaves as enabled. You can check virsh domcapabilities to see if QEMU/KVM is actually able to enable xsaves on this host or not. > Login to the guest, cat /proc/cpuinfo, there is no xsaves. This is correct since QEMU did not enable xsaves for the virtual CPU. > 3. Check the feature xsaves is not support migration > # cat /usr/share/libvirt/cpu_map.xml|grep xsaves > <feature name='xsaves' migratable='no'> This is not used with QEMU >= 2.9.0 since we get the migrtability data directly from QEMU, but it doesn't make any difference here. > 4. Do migration without any error message, should failed with error message, > as this cpu feature is not migratable. No, we detected xsaves is not actually enabled in the virtual CPU so there's no reason to block the migration. > 1) managedsave: after managedsave, start guest, the xsaves=off on the qemu > command line. > # virsh managedsave r7-4t > Domain r7-4t state saved by libvirt > > # virsh start r7-4t > Domain r7-4t started > > # virsh dumpxml r7-4t|grep xsaves > <feature policy='disable' name='xsaves'/> > > # ps -ef|grep qemu| grep r7-4t| grep xsaves > qemu 10127 1 0 01:44 ? 00:00:00 /usr/libexec/qemu-kvm -name > guest=r7-4t,debug-threads=on -S -object > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-56-r7-4t/ > master-key.aes -machine > pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off -cpu > Skylake-Client,ss=on,hypervisor=on,tsc_adjust=on,clflushopt=on,pdpe1gb=on, > xsaves=off..... This is correct, we already no the xsaves was disabled so we don't enable it when restoring the domain to maintain guest ABI. > Login to the guest, cat /proc/cpuinfo, there is no xsaves. Correct. > 2)migration: no error message. > # virsh migrate r7-4t qemu+ssh://10.73.73.83/system --live --verbose > Migration: [100 %] Correct. > Additional info: > 1. Met the same issues when test with cpu mode custom and with xsaves > required. > > xml: > <cpu mode='custom' match='exact'> > <model fallback='forbid'>Skylake-Client</model> > <feature policy='require' name='xsaves'/> > </cpu> > > # virsh start r7-4t > Domain r7-4t started > > # virsh dumpxml r7-4t|grep "<cpu" -A 9 > <cpu mode='custom' match='exact' check='full'> > <model fallback='forbid'>Skylake-Client</model> > <feature policy='disable' name='xsaves'/> > <feature policy='require' name='hypervisor'/> > </cpu> Yeah, this is similar to host-model case. You asked for xsaves, but QEMU did not enable it. So far it looks like everything works as expected. Whether QEMU was supposed to disable xsaves on the host or not is separate question, though. Could you check that xsaves is really not supported by QEMU/KVM on that host by starting QEMU directly (without libvirt) with -cpu host,migratable=off and checking whether the feature is enabled in the guest or not. Or alternatively, remove all cache files from /var/cache/libvirt/qemu, restart libvirtd with debug messages enabled and attach the debug logs generated when libvirtd started. (In reply to Jiri Denemark from comment #14) > (In reply to chhu from comment #11) > > 1. Start a VM with cpu featrue: xsaves, the policy will changed to disable > > in the dumpxml, but the qemu command line shows: xsaves=on, login to the > > guest, cat /proc/cpuinfo, there is no xsaves. > > > > xml: > > <cpu mode='host-model'> > > <model fallback='allow'/> > > <feature policy='require' name='xsaves'/> > > </cpu> > > This is asking for a host-model CPU (which can be seen in virsh > domcapabilities) and trying to force xsaves. Since the host CPUID claims > xsaves is present, libvirt won't complain about this. Adding check='full' > would change this, libvirt would refuse the start the domain. > The test result is as what your said. Thank you! Test with packages: libvirt-3.2.0-14.el7.x86_64 qemu-kvm-rhev-2.9.0-12.el7.x86_64 Test steps: 1. On the host not support xsaves but cpuid shows the XSAVES is enabled, try to start a guest with cpu feature:xsaves with check=full, libvirt refuse to start the doman: PASS # virsh dumpxml r7-4t| grep '<cpu' -A 5 <cpu mode='host-model' check='full'> <model fallback='allow'/> <feature policy='require' name='xsaves'/> </cpu> # virsh start r7-4t error: Failed to start domain r7-4t error: operation failed: guest CPU doesn't match specification: missing features: xsaves > > # virsh start r7-4t > > Domain r7-4t started > > > > # virsh dumpxml r7-4t| grep "<cpu" -A 12 > > <cpu mode='custom' match='exact' check='full'> > > <model fallback='forbid'>Skylake-Client</model> > > <vendor>Intel</vendor> > > <feature policy='require' name='ss'/> > > <feature policy='require' name='hypervisor'/> > > <feature policy='require' name='tsc_adjust'/> > > <feature policy='require' name='clflushopt'/> > > <feature policy='require' name='pdpe1gb'/> > > <feature policy='disable' name='xsaves'/> > > </cpu> > > So this means QEMU/KVM disabled xsave even though the CPUID claims the > features is present and you explicitly requested the feature to be turned on. > QEMU is allowed to do this since you didn't use check='full' when starting > the > domain. > > > # ps -ef|grep qemu| grep r7-4t| grep xsaves > > qemu 8353 1 5 01:20 ? 00:00:15 /usr/libexec/qemu-kvm -name > > guest=r7-4t,debug-threads=on -S -object > > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-55-r7-4t/ > > master-key.aes -machine > > pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off -cpu > > Skylake-Client,ss=on,hypervisor=on,tsc_adjust=on,clflushopt=on,pdpe1gb=on, > > xsaves=on ...... > > This just reflects the request to enable the feature, it doesn't mean QEMU > will actually enable it. I believe you should see a warning about it in > /var/log/libvirt/qemu/r7-4t.log Yes, there is a warning. ----------------------------------------------------------------------------- On the host not support xsaves but cpuid shows the XSAVES is enabled, try to start a guest with cpu feature: xsaves without check=full, libvirt start the guest, and check in /var/log/libvirt/qemu/r7-4t.log, there is a warining: warning: host doesn't support requested feature: CPUID.0DH:EAX.xsaves [bit 3] ----------------------------------------------------------------------------- > > > 2. On host: cat /proc/cpuinfo|grep xsaves, no output, not include xsaves. > > But host cpuid shows the XSAVES is enabled, and virsh capabilites include > > xsaves. > > # cpuid > > ...... > > XSAVE features (0xd/1): > > XSAVEOPT instruction = true > > XSAVEC instruction = true > > XGETBV instruction = true > > XSAVES/XRSTORS instructions = true > > SAVE area size in bytes = 0x000003c0 (960) > > IA32_XSS lower 32 bits valid bit field mask = 0x00000100 > > IA32_XSS upper 32 bits valid bit field mask = 0x00000000 > > > > # virsh capabilities|grep xsaves > > <feature name='xsaves'/> > > This is just saying host's CPUID shows xsaves as enabled. You can check virsh > domcapabilities to see if QEMU/KVM is actually able to enable xsaves on this > host or not. Check the virsh domcapabilities, xsaves is not in the list. # virsh domcapabilities <cpu> <mode name='host-passthrough' supported='yes'/> <mode name='host-model' supported='yes'> <model fallback='forbid'>Skylake-Client</model> <vendor>Intel</vendor> <feature policy='require' name='ss'/> <feature policy='require' name='hypervisor'/> <feature policy='require' name='tsc_adjust'/> <feature policy='require' name='clflushopt'/> <feature policy='require' name='pdpe1gb'/> <feature policy='require' name='invtsc'/> </mode> <mode name='custom' supported='yes'> ...... <model usable='yes'>Skylake-Client</model> ...... </mode> </cpu> > > > Login to the guest, cat /proc/cpuinfo, there is no xsaves. > > This is correct since QEMU did not enable xsaves for the virtual CPU. > > > 3. Check the feature xsaves is not support migration > > # cat /usr/share/libvirt/cpu_map.xml|grep xsaves > > <feature name='xsaves' migratable='no'> > > This is not used with QEMU >= 2.9.0 since we get the migrtability data > directly from QEMU, but it doesn't make any difference here. > > > 4. Do migration without any error message, should failed with error message, > > as this cpu feature is not migratable. > > No, we detected xsaves is not actually enabled in the virtual CPU so there's > no reason to block the migration. Got them, thank you! > > > 1) managedsave: after managedsave, start guest, the xsaves=off on the qemu > > command line. > > # virsh managedsave r7-4t > > Domain r7-4t state saved by libvirt > > > > # virsh start r7-4t > > Domain r7-4t started > > > > # virsh dumpxml r7-4t|grep xsaves > > <feature policy='disable' name='xsaves'/> > > > > # ps -ef|grep qemu| grep r7-4t| grep xsaves > > qemu 10127 1 0 01:44 ? 00:00:00 /usr/libexec/qemu-kvm -name > > guest=r7-4t,debug-threads=on -S -object > > secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-56-r7-4t/ > > master-key.aes -machine > > pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off -cpu > > Skylake-Client,ss=on,hypervisor=on,tsc_adjust=on,clflushopt=on,pdpe1gb=on, > > xsaves=off..... > > This is correct, we already no the xsaves was disabled so we don't enable it > when restoring the domain to maintain guest ABI. > > > Login to the guest, cat /proc/cpuinfo, there is no xsaves. > > Correct. > > > 2)migration: no error message. > > # virsh migrate r7-4t qemu+ssh://10.73.73.83/system --live --verbose > > Migration: [100 %] > > Correct. > Got them, thank you! > > Additional info: > > 1. Met the same issues when test with cpu mode custom and with xsaves > > required. > > > > xml: > > <cpu mode='custom' match='exact'> > > <model fallback='forbid'>Skylake-Client</model> > > <feature policy='require' name='xsaves'/> > > </cpu> > > > > # virsh start r7-4t > > Domain r7-4t started > > > > # virsh dumpxml r7-4t|grep "<cpu" -A 9 > > <cpu mode='custom' match='exact' check='full'> > > <model fallback='forbid'>Skylake-Client</model> > > <feature policy='disable' name='xsaves'/> > > <feature policy='require' name='hypervisor'/> > > </cpu> > > Yeah, this is similar to host-model case. You asked for xsaves, but QEMU did > not enable it. > Yes, Got the same test results as hot-model case: PASS 1. On the host not support xsaves but cpuid shows the XSAVES is enabled, try to start a guest with cpu feature:xsaves with check=full, libvirt refuse to start the doman: PASS # virsh dumpxml r7-4t| grep '<cpu' -A 5 <cpu mode='custom' match='exact' check='full'> <model fallback='forbid'>Skylake-Client</model> <feature policy='require' name='xsaves'/> <feature policy='require' name='hypervisor'/> </cpu> # virsh start r7-4t error: Failed to start domain r7-4t error: operation failed: guest CPU doesn't match specification: missing features: xsaves 2. On the host not support xsaves but cpuid shows the XSAVES is enabled, try to start a guest with custom mode with cpu feature: xsaves without check=full, libvirt start the guest, and check in /var/log/libvirt/qemu/r7-4t.log, there is a warining: warning: host doesn't support requested feature: CPUID.0DH:EAX.xsaves [bit 3] # virsh dumpxml r7-4t| grep '<cpu' -A 5 <cpu mode='custom' match='exact' check='full'> <model fallback='forbid'>Skylake-Client</model> <feature policy='disable' name='xsaves'/> <feature policy='require' name='hypervisor'/> </cpu> > So far it looks like everything works as expected. Whether QEMU was supposed
> to disable xsaves on the host or not is separate question, though. Could you
> check that xsaves is really not supported by QEMU/KVM on that host by
> starting
> QEMU directly (without libvirt) with -cpu host,migratable=off and checking
> whether the feature is enabled in the guest or not. Or alternatively, remove
> all cache files from /var/cache/libvirt/qemu, restart libvirtd with debug
> messages enabled and attach the debug logs generated when libvirtd started.
Checked the xsaves is not supported by QEMU/KVM on this host:
Removed all cache files from /var/chache/libvirt/qemu, restart libvirtd with debug messages enabled.
1. Check in the qemu/capabilities file: xsaves with value false.
For x86_64:
-------------------------------------------------------------
<hostCPU type='kvm' model='base' migratability='yes'>
......
381 <property name='xsaves' type='boolean' value='false'/>
-------------------------------------------------------------
More details:
# cat /var/cache/libvirt/qemu/capabilities/3c76bc41d59c0c7314b1ae8e63f4f765d2cf16abaeea081b3ca1f5d8732f7bb1.xml
1 <qemuCaps>
......
207 <kvmVersion>0</kvmVersion>
208 <package>(qemu-kvm-rhev-2.9.0-12.el7)</package>
209 <arch>x86_64</arch>
210 <hostCPU type='kvm' model='base' migratability='yes'>
211 <property name='phys-bits' type='number' value='0'/>
......
381 <property name='xsaves' type='boolean' value='false'/>
......
437 <hostCPU type='tcg' model='base' migratability='yes'>
......
608 <property name='xsaves' type='boolean' value='false'/>
663 </hostCPU>
......
744 </qemuCaps>
2. Check in libvirtd.log: "xsaves": false
More details:
info : qemuMonitorIOProcess:431 : QEMU_MONITOR_IO_PROCESS: mon=0x7efeac1df740 buf={"return": {"model": {"name": "base", "props": {"cmov": true, "ia64": false, "aes": true, "mmx": true,...... "xsaves": false, "lm": true......
debug : qemuMonitorJSONIOProcessLine:193 : Line [{"return": {"model": {"name": "base", "props": {"cmov": true, "ia64": false, "aes": true, "mmx": true, "rdpid": false, "arat": true......"xsaves": false, "lm": true,......
debug : virJSONValueFromString:1713 : string={"return": {"model": {"name": "base", "props": {"cmov": true, "ia64": false, "aes": true, "mmx": true, "rdpid": false, "arat": true,......"xsaves": false, "lm": true,......
info : qemuMonitorJSONIOProcessLine:213 : QEMU_MONITOR_RECV_REPLY: mon=0x7efeac1df740 reply={"return": {"model": {"name": "base", "props": {"cmov": true, "ia64": false, "aes": true, "mmx": true,......"xsaves": false, "lm": true,......
debug : virJSONValueToString:1904 : result={"execute":"query-cpu-model-expansion","arguments":{"type":"full","model":{"name":"base","props":{"cmov":true,"ia64":false,"aes":true,"mmx":true,......"xsaves":false,"lm":true,......
debug : qemuMonitorJSONCommandWithFd:298 : Send command '{"execute":"query-cpu-model-expansion","arguments":{"type":"full","model":{"name":"base","props":{"cmov":true,"ia64":false,"aes":true,"mmx":true,......"xsaves":false,"lm":true,......
info : qemuMonitorSend:1034 : QEMU_MONITOR_SEND_MSG: mon=0x7efeac1df740 msg={"execute":"query-cpu-model-expansion","arguments":{"type":"full","model":{"name":"base","props":{"cmov":true,"ia64":false,"aes":true,"mmx":true,"rdpid":false,"arat":true,......
"xsaves":false,"lm":true,......
info : qemuMonitorIOWrite:536 : QEMU_MONITOR_IO_WRITE: mon=0x7efeac1df740 buf={"execute":"query-cpu-model-expansion","arguments":{"type":"full","model":{"name":"base","props":{"cmov":true,"ia64":false,"aes":true,"mmx":true,"rdpid":false,"arat":true,......
"xsaves":false,"lm":true,......
1. (In reply to chhu from comment #7) > Add another issue related to non-migratable features: > > Imporve the error message: > Migrate with 'invtsc' get qemu error: > "unable to execute QEMU command 'migrate': State blocked by non-migratable > device 'cpu'" > > The error message should be reported by libvirt as: > "error: Requested operation is not valid: domain has CPU feature: invtsc" > > As the xsaves is also in the qemu-kvm command line, the error message should > also include xsaves. > > How reproducible: > 100% > > Version-Release number of selected component (if applicable): > qemu-kvm-rhev-2.6.0-22.el7.x86_64 > libvirt-2.0.0-6.el7.x86_64 > kernel: kernel-3.10.0-495.el7.x86_64 > > Steps to reproduce: > 1. Start a VM with xml below successfully. > # virsh start r7-s1 > Domain r7-s1 started > > # virsh dumpxml r7-s1|grep "<cpu" -A 6 > <cpu mode='custom' match='minimum'> > <model fallback='forbid'>Skylake-Client</model> > </cpu> > > # ps -ef|grep r7-s1 > qemu 17490 1 11 22:59 ? 00:00:12 /usr/libexec/qemu-kvm -name > guest=r7-s1...... -cpu > Skylake-Client,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,+monitor,+ds_cpl,+vmx,+smx, > +est,+tm2,+xtpr,+pdcm,+osxsave,+tsc_adjust,+clflushopt,+xsaves,+pdpe1gb, > +invtsc -m 1024 > > 2. Try to do migration test: met error: "unable to execute QEMU command > 'migrate': State blocked by non-migratable device 'cpu'" > > # virsh migrate r7-s1 qemu+ssh://10.19.176.166/system --live --verbose > root.176.166's password: > error: internal error: unable to execute QEMU command 'migrate': State > blocked by non-migratable device 'cpu' Tested on packages: libvirt-3.2.0-14.el7.x86_64 qemu-kvm-rhev-2.9.0-12.el7.x86_64 Test steps: 1. Define and start a guest with xml below: <cpu mode='custom' match='minimum'> <model fallback='forbid'>Skylake-Client</model> </cpu> # virsh list --all Id Name State ---------------------------------------------------- 45 r7-4t running 2. Check the xsaves is not include in qemu command line. As the host not support xsaves: PASS # virsh dumpxml r7-4t| grep '<cpu' -A 7 <cpu mode='custom' match='exact' check='full'> <model fallback='forbid'>Skylake-Client</model> <feature policy='require' name='ss'/> <feature policy='require' name='hypervisor'/> <feature policy='require' name='tsc_adjust'/> <feature policy='require' name='clflushopt'/> <feature policy='require' name='pdpe1gb'/> </cpu> # ps -ef|grep r7-4t qemu 32521 1 2 05:15 ? 00:00:16 /usr/libexec/qemu-kvm -name guest=r7-4t,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-45-r7-4t/master-key.aes -machine pc-i440fx-rhel7.4.0,accel=kvm,usb=off,dump-guest-core=off -cpu Skylake-Client,ss=on,hypervisor=on,tsc_adjust=on,clflushopt=on,pdpe1gb=on -m 1024 -realtime mlock=off -smp 4,sockets=4,cores=1,threads=1...... (In reply to chhu from comment #18) > 1. (In reply to chhu from comment #7) > > Add another issue related to non-migratable features: > > > > Imporve the error message: > > Migrate with 'invtsc' get qemu error: > > "unable to execute QEMU command 'migrate': State blocked by non-migratable > > device 'cpu'" > > > > The error message should be reported by libvirt as: > > "error: Requested operation is not valid: domain has CPU feature: invtsc" > > Tested on packages: libvirt-3.2.0-14.el7.x86_64 qemu-kvm-rhev-2.9.0-12.el7.x86_64 Test steps: 1. Define and start a guest with xml below: <cpu mode='custom' match='exact' check='full'> <model fallback='forbid'>Skylake-Client</model> <feature policy='require' name='invtsc'/> <feature policy='require' name='hypervisor'/> </cpu> # virsh start r7-4t Domain r7-4t started 2. Try to migrate with invstc, failed with error message: PASS checked the invtsc is not migratable: <hostCPU type='kvm' model='base' migratability='yes'> <property name='phys-bits' type='number' value='0'/> ...... <property name='invtsc' type='boolean' value='true' migratable='no'/> ...... # virsh migrate r7-4t qemu+ssh://10.73.73.83/system --live --verbose error: Requested operation is not valid: domain has 'invtsc' CPU feature but TSC frequency is not specified Thanks Jiri's detailed reply!! According to the comment 15,16,17,18,19, the bug is verified, set the status to VERIFIED. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1846 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1846 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2017:1846 |
Description: CPU feature xsaves is not work properly. How reproducible: 100% Version-Release number of selected component (if applicable): qemu-kvm-rhev-2.6.0-22.el7.x86_64 libvirt-2.0.0-6.el7.x86_64 Steps to reproduce: 1. Start a VM with cpu feature: xsaves. # virsh start rhel7 Domain rhel7 started # virsh dumpxml rhel7| grep "<cpu" -A 6 <cpu mode='host-model'> <model fallback='allow'/> <feature policy='require' name='xsaves'/> </cpu> # ps -ef|grep qemu qemu 1384 1 29 04:10 ? 00:00:12 /usr/libexec/qemu-kvm -name guest=rhel7... -cpu Skylake-Client,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,+monitor,+ds_cpl,+vmx,+smx,+est,+tm2, +xtpr,+pdcm,+osxsave,+tsc_adjust,+clflushopt,+xsaves,+pdpe1gb -m 1024... 2. On host: cat /proc/cpuinfo | grep xsaves, no output, not include xsaves. but host cpuid shows the XSAVES is enabled, and virsh capabilites, include xsaves. # cpuid XSAVE features (0xd/1): XSAVEOPT instruction = true XSAVEC instruction = true XGETBV instruction = true XSAVES/XRSTORS instructions = true SAVE area size in bytes = 0x000003c0 (960) IA32_XSS lower 32 bits valid bit field mask = 0x00000100 IA32_XSS upper 32 bits valid bit field mask = 0x00000000 # virsh capabilities|grep xsaves <feature name='xsaves'/> Login to the guest, cat /proc/cpuinfo, there is no xsaves. 3. Check the feature xsaves is not support migration # cat /usr/share/libvirt/cpu_map.xml|grep xsaves <feature name='xsaves' migratable='no'> # virsh capabilities >capabilities.xml # virsh cpu-baseline capabilities.xml | grep xsaves <feature policy='require' name='xsaves'/> # virsh cpu-baseline capabilities.xml --migratable| grep xsaves <no output> 4. Do managedsave and migrate without any error message, the operations should failed with error message, similar as: "error: Requested operation is not valid: domain has CPU feature xsaves" 1) managedsave: no error message # virsh managedsave rhel7 Domain rhel7 state saved by libvirt # virsh start rhel7 Domain rhel7 started # virsh dumpxml rhel7| grep xsaves <feature policy='require' name='xsaves'/> # ps -ef|grep qemu qemu 1703 1 0 04:38 ? 00:00:00 /usr/libexec/qemu-kvm -name guest=rhel7... -cpu Skylake-Client,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,+monitor,+ds_cpl,+vmx, +smx,+est,+tm2,+xtpr,+pdcm,+osxsave,+tsc_adjust,+clflushopt,+xsaves,+pdpe1gb 2) migration: no error message. # virsh migrate rhel7 qemu+ssh://ipaddress/system --live --verbose Migration: [100 %] Actual results: In step2: the guest start successfully, but the guest cpu not support xsaves. In step4: do managedsave and migrate, no error message. Expected results: In step2: start the guest should fail, or the guest cpu should support xsaves. In step4: do managedsave and migrate, should failed with error message, as this cpu feature is not migratable.