Bug 1377913

Summary:	Libvirt should refuse to start a Skylake-Client VM on a non-Skylake CPU.
Product:	Red Hat Enterprise Linux 7	Reporter:	chhu
Component:	libvirt	Assignee:	Jiri Denemark <jdenemar>
Status:	CLOSED ERRATA	QA Contact:	Jing Qi <jinqi>
Severity:	medium	Docs Contact:
Priority:	unspecified
Version:	7.3	CC:	chhu, dyuan, jdenemar, lhuang, rbalakri, xuzhang, yalzhang
Target Milestone:	rc
Target Release:	7.4
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	libvirt-2.5.0-1.el7	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-08-01 17:16:43 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	1199452

Description chhu 2016-09-21 02:00:25 UTC

Description: libvirt should refuse to start a Skylake-Client VM on a non-Skylake CPU.

How producible:
100%

Tested with packages:
libvirt-2.0.0-9.el7.x86_64
qemu-kvm-rhev-2.6.0-25.el7.x86_64
3.10.0-506.el7.x86_64

Test steps:
1. Started a VM with host mode: Skylake-Client on host: Haswell-noTSX/Broadwell/SandyBridge successfully. Should refuse to start the VM.
<cpu mode='host-model'>
 <model>Skylake-Client</model> or <model fallback='forbid'>Skylake-Client</model>
</cpu>

1)on host: Haswell-noTSX
# virsh start r7.3
Domain r7.3 started

# ps -ef|grep qemu-kvm
qemu      19302      1  4 22:07 ?        00:00:21 /usr/libexec/qemu-kvm -name guest=r7.3... -cpu Haswell-noTSX,+vme,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,
+monitor,+ds_cpl,+vmx,+smx,+est,+tm2,+xtpr,+pdcm,+dca,+osxsave,+f16c,+rdrand,
+arat,+tsc_adjust,+xsaveopt,+pdpe1gb,+abm -m 1024

2)on host: Broadwell
# virsh start r7.3
Domain r7.3 started

# ps -ef|grep qemu
qemu      22036      1 40 01:00 ?        00:00:20 /usr/libexec/qemu-kvm -name guest=r7.3,...-cpu Broadwell,+vme,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,
+monitor,+ds_cpl,+vmx,+smx,+est,+tm2,+xtpr,+pdcm,+dca,+osxsave,+f16c,+rdrand,
+arat,+tsc_adjust,+xsaveopt,+pdpe1gb,+abm,+rtm,+hle -m 1024

3)on host: SandyBridge
# virsh start r7.3
Domain r7.3 started

# ps -ef|grep qemu
qemu      4326     1 70 15:04 ?        00:00:20 /usr/libexec/qemu-kvm -name guest=r7.3,... -cpu SandyBridge,+vme,+ds,+acpi,+ss,+ht,+tm,+pbe,+dtes64,
+monitor,+ds_cpl,+vmx,+smx,+est,+tm2,+xtpr,+pdcm,+pcid,+osxsave,+arat,
+xsaveopt -m 1024

Actual results:
Start a Skylake-Client VM on a non-Skylake CPU:Haswell-noTSX/Broadwell/SandyBridge successfully

Expected results:
Libvirt should refuse to start a Skylake-Client VM on a non-Skylake CPU.

Comment 4 Jiri Denemark 2016-09-22 13:56:59 UTC

This should be fixed upstream as of

commit 7ce711a30eaf882ccd0217b2528362b563b6d670
Refs: v2.2.0-199-g7ce711a
Author:     Jiri Denemark <jdenemar>
AuthorDate: Wed Jun 22 15:53:48 2016 +0200
Commit:     Jiri Denemark <jdenemar>
CommitDate: Thu Sep 22 15:40:09 2016 +0200

    qemu: Update guest CPU def in live XML

    Storing the updated CPU definition in the live domain definition saves
    us from having to update it over and over when we need it. Not to
    mention that we will soon further update the CPU definition according to
    QEMU once it's started.

    A highly wanted side effect of this patch, libvirt will pass all CPU
    features explicitly specified in domain XML to QEMU, even those that are
    already included in the host model.

    This patch should fix the following bugs:
        https://bugzilla.redhat.com/show_bug.cgi?id=1207095
        https://bugzilla.redhat.com/show_bug.cgi?id=1339680
        https://bugzilla.redhat.com/show_bug.cgi?id=1371039
        https://bugzilla.redhat.com/show_bug.cgi?id=1373849
        https://bugzilla.redhat.com/show_bug.cgi?id=1375524
        https://bugzilla.redhat.com/show_bug.cgi?id=1377913

    Signed-off-by: Jiri Denemark <jdenemar>

Comment 6 Jing Qi 2017-03-03 07:12:15 UTC

Verified with version: libvirt-2.5.0-1.el7.x86_64
                       qemu-kvm-rhev-2.8.0-3.el7.x86_64

On a host with CPU type of "SandyBridge" and the cpu part in domain xml file is as below, the guest vm can't be started.

<cpu mode='host-model'>
    <model fallback='allow'>Skylake-Client</model>
  </cpu>

#virsh start  rhel7
error: Failed to start domain rhel7
error: the CPU is incompatible with host CPU: Host CPU does not provide required features: fma, movbe, f16c, rdrand, fsgsbase, bmi1, hle, avx2, smep, bmi2, erms, invpcid, rtm, mpx, rdseed, adx, smap, xsavec, xgetbv1, abm, 3dnowprefetch

Comment 7 yalzhang@redhat.com 2017-06-18 07:10:00 UTC

Hi Jiri,

I have a question about this bug. In libvirt.org page, it is said that for <cpu mode='host-model'> :

"The match attribute can't be used in this mode. Specifying CPU model is not supported either, but model's fallback attribute may still be used."

Refer to: http://libvirt.org/formatdomain.html#elementsCPU

As it is described in comment 0, the guest started but with the host-model cpu(Haswell-noTSX/Broadwell/SandyBridge), not Skylake-Client which is specified in the xml. I think it is acceptable, and libvirt just need to ignore the model setting in the xml:

<model>Skylake-Client</model> or <model fallback='forbid'>Skylake-Client</model>

or report error to such setting such as "Specifying CPU model is not supported in <cpu mode='host-model'>"

If the customer want to use Skylake-Client, he need to set like this:

<cpu mode='custom'>
 <model>Skylake-Client</model>
</cpu>

Could you help to confirm? thank you very much!

Comment 8 Jiri Denemark 2017-06-19 08:41:17 UTC

(In reply to yalzhang from comment #7)
> As it is described in comment 0, the guest started but with the host-model
> cpu(Haswell-noTSX/Broadwell/SandyBridge), not Skylake-Client which is
> specified in the xml. I think it is acceptable, and libvirt just need to
> ignore the model setting in the xml:
> 
> <model>Skylake-Client</model> or <model
> fallback='forbid'>Skylake-Client</model>
> 
> or report error to such setting such as "Specifying CPU model is not
> supported in <cpu mode='host-model'>"

No, we need to ignore it to maintain backward compatibility. Reporting an error could break existing domains.

> If the customer want to use Skylake-Client, he need to set like this:
> 
> <cpu mode='custom'>
>  <model>Skylake-Client</model>
> </cpu>

Yes.

Comment 9 yalzhang@redhat.com 2017-06-21 07:55:25 UTC

(In reply to Jiri Denemark from comment #8)
> (In reply to yalzhang from comment #7)
> > As it is described in comment 0, the guest started but with the host-model
> > cpu(Haswell-noTSX/Broadwell/SandyBridge), not Skylake-Client which is
> > specified in the xml. I think it is acceptable, and libvirt just need to
> > ignore the model setting in the xml:
> > 
> > <model>Skylake-Client</model> or <model
> > fallback='forbid'>Skylake-Client</model>
> > 

How do you think about this proposal? 

As it is said in libvirt.org:
"The host-model mode is essentially a shortcut to copying host CPU definition from capabilities XML into domain XML. Since the CPU definition is copied just before starting a domain, ..." and "Specifying CPU model is not supported either..." 

As in comment 6, libvirt check the model and fails, but in fact, the model will not be used in the running guest. So the checking just makes no sense.

For PowerISA, it seems different, may be current behavior is needed for the compatibility or the document need to be modified. Please help to confirm, thank you!

Refer to http://libvirt.org/formatdomain.html#elementsCPU  the "host-model" part
there is an example like this:
<cpu mode='host-model'>
  <model>power7</model>
</cpu>

Comment 10 Jiri Denemark 2017-06-21 11:25:36 UTC

The documentation already says power is different: "PowerISA allows processors to run VMs in binary compatibility mode supporting an older version of ISA. Libvirt on PowerPC architecture uses the host-model to signify a guest mode CPU running in binary compatibility mode. Example: When a user needs a power7 VM to run in compatibility mode on a Power8 host, this can be described in XML as follows : 

<cpu mode='host-model'>
  <model>power7</model>
</cpu>"

Comment 11 yalzhang@redhat.com 2017-06-26 03:01:31 UTC

Hi, Jiri, sorry for that I did not make it clear. Please help to confirm, thank you very much.

As it is described in comment 0, the guest started but with the host-model cpu(Haswell-noTSX/Broadwell/SandyBridge), not Skylake-Client which is specified in the xml. 

In addition，
As it is said in libvirt.org:
"The host-model mode is essentially a shortcut to copying host CPU definition from capabilities XML into domain XML. Since the CPU definition is copied just before starting a domain, ..." and "Specifying CPU model is not supported either..." 

As in comment 6, libvirt check the model and fails, but in fact, the model will not be used in the running guest. So the checking just makes no sense.

I think libvirt may need to only ignore the model setting in the xml(may be only for x86_64 model, not for power).

Expected:
When set host-model with skylake-client model like this:
 <cpu mode='host-model'>
    <model fallback='allow'>Skylake-Client</model>
  </cpu>

libvirt ignore the model setting in inactive xml
 <cpu mode='host-model'/>

and the guest can start with the cpu xml copy from "# virsh domcapabilites"

Comment 12 Jiri Denemark 2017-06-27 07:59:03 UTC

I somehow didn't notice the bugs description was talking about <cpu mode='host-model'><model>Skylake-Client</model></cpu>. This is a valid configuration sent by old libvirt during migration (new libvirt transforms host-model into custom CPU), but it's not something users should add into the XML. Libvirt will honor the CPU model for incoming migration, but it may just ignore it when starting a fresh domain. This applies to x86_64, of course. On Power, the model in a host-model CPU will always be honored.

Comment 13 errata-xmlrpc 2017-08-01 17:16:43 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846

Comment 14 errata-xmlrpc 2017-08-01 23:57:38 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846