Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 824989

Summary:	make sure guest CPU doesn't change when a domain is started on another host
Product:	Red Hat Enterprise Linux 7	Reporter:	Eduardo Habkost <ehabkost>
Component:	libvirt	Assignee:	Jiri Denemark <jdenemar>
Status:	CLOSED ERRATA	QA Contact:	Luyao Huang <lhuang>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	7.1	CC:	cwei, dyuan, jdenemar, jsuchane, knoel, lhuang, mzhan, tburke, xuzhang, yalzhang
Target Milestone:	rc	Keywords:	Upstream
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:	libvirt-3.2.0-1.el7	Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-08-01 17:06:41 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	824987
Bug Blocks:	725080, 822148, 922719, 1018251, 1199452, 1313485

Description Eduardo Habkost 2012-05-24 18:28:55 UTC

libvirt needs to query QEMU for CPU capabilities, to know which CPU features may be enabled (based on host CPU + kernel + QEMU capabilities), and to know which features are included on each preconfigured CPU model inside QEMU.

Today libvirt queries the host CPU directly using the CPUID instruction. It have mostly worked until now because often features were included on libvirt only after the kernel and QEMU were changed to support them. The current method breaks if the kernel or QEMU needs to be updated to include support for new features, or if a feature is completedly emulated by QEMU or KVM (e.g. x2apic).

Comment 1 Eduardo Habkost 2013-05-14 17:16:48 UTC

Reference for how libvirt could use the feature from bug 824987:
http://wiki.qemu.org/Features/CPUModels#Interfaces.2Frequirements_for_libvirt

Comment 10 Jiri Denemark 2017-03-14 16:59:40 UTC

Patches sent upstream for review: https://www.redhat.com/archives/libvir-list/2017-March/msg00578.html

Comment 11 Jiri Denemark 2017-03-17 11:02:36 UTC

Finally fixed upstream by:

commit cab2c1af96c4b446e7f0fb72809402302a2560be
Refs: v3.1.0-182-gcab2c1af9
Author:     Jiri Denemark <jdenemar>
AuthorDate: Wed Mar 1 14:54:53 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 17 11:50:47 2017 +0100

    tests: Switch to sparse initialization of virCPUDef

    Signed-off-by: Jiri Denemark <jdenemar>

commit 38567e94dea56eeafa487d754875cdc6a5e776d9
Refs: v3.1.0-183-g38567e94d
Author:     Jiri Denemark <jdenemar>
AuthorDate: Thu Mar 2 14:53:18 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 17 11:50:48 2017 +0100

    docs: Clarify /domain/cpu/@match description

    Signed-off-by: Jiri Denemark <jdenemar>

commit 641b8c721e01dd2a7d0cae7b7f3b7a06e43f9da2
Refs: v3.1.0-184-g641b8c721
Author:     Jiri Denemark <jdenemar>
AuthorDate: Wed Mar 1 15:18:22 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 17 11:50:48 2017 +0100

    Introduce /domain/cpu/@check XML attribute

    The attribute can be used to request a specific way of checking whether
    the virtual CPU matches created by the hypervisor matches the
    specification in domain XML.

    Signed-off-by: Jiri Denemark <jdenemar>

commit fcd56ce86640bf8bc230025a3791593a5bf10507
Refs: v3.1.0-185-gfcd56ce86
Author:     Jiri Denemark <jdenemar>
AuthorDate: Wed Mar 1 16:12:07 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 17 11:50:48 2017 +0100

    qemu: Set default values for CPU check attribute

    Signed-off-by: Jiri Denemark <jdenemar>

commit e9dbe70110ac2d6ad8264c1c404e5ca4069e0e7b
Refs: v3.1.0-186-ge9dbe7011
Author:     Jiri Denemark <jdenemar>
AuthorDate: Fri Mar 10 23:55:59 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 17 11:50:48 2017 +0100

    qemu: Refactor Hyper-V features check

    The checks are now in a dedicated qemuProcessVerifyHypervFeatures
    function.

    In addition to moving the code this patch also fixes a few bugs: the
    original code was leaking cpuFeature and the return value of
    virCPUDataCheckFeature was not checked properly.

    Signed-off-by: Jiri Denemark <jdenemar>

commit d5f47d7d75eefe6afd941f963515cafe865c7ab0
Refs: v3.1.0-187-gd5f47d7d7
Author:     Jiri Denemark <jdenemar>
AuthorDate: Sat Mar 11 00:10:56 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 17 11:50:48 2017 +0100

    qemu: Refactor KVM features check

    The checks are now in a dedicated qemuProcessVerifyKVMFeatures function.

    Signed-off-by: Jiri Denemark <jdenemar>

commit af1ca855454e026395ac1e6f515381ed2f5aa0c4
Refs: v3.1.0-188-gaf1ca8554
Author:     Jiri Denemark <jdenemar>
AuthorDate: Sat Mar 11 00:36:55 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 17 11:50:48 2017 +0100

    qemu: Refactor CPU features check

    The checks are now in a dedicated qemuProcessVerifyCPUFeatures function.

    Signed-off-by: Jiri Denemark <jdenemar>

commit cfeee3373bf2afefe7bbf958af6c00e672756f20
Refs: v3.1.0-189-gcfeee3373
Author:     Jiri Denemark <jdenemar>
AuthorDate: Sat Mar 11 00:45:37 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 17 11:50:48 2017 +0100

    qemu: Refactor qemuProcessVerifyGuestCPU

    Signed-off-by: Jiri Denemark <jdenemar>

commit 253db85e2d3da7e997dc4860c0009bdb7afbef4c
Refs: v3.1.0-190-g253db85e2
Author:     Jiri Denemark <jdenemar>
AuthorDate: Mon Mar 13 10:23:37 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 17 11:50:48 2017 +0100

    qemu: Use ARCH_IS_X86 in qemuMonitorJSONGetGuestCPU

    Signed-off-by: Jiri Denemark <jdenemar>

commit 77c9c4f127e983f67561fafcb257fb4c440bd28f
Refs: v3.1.0-191-g77c9c4f12
Author:     Jiri Denemark <jdenemar>
AuthorDate: Mon Mar 13 11:00:48 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 17 11:50:48 2017 +0100

    qemu: Ask QEMU for filtered CPU features

    qemuMonitorGetGuestCPU can now optionally create CPU data from
    filtered-features in addition to feature-words.

    Signed-off-by: Jiri Denemark <jdenemar>

commit def9401acb6f1b6a4eefbb8b5c84629811bc309d
Refs: v3.1.0-192-gdef9401ac
Author:     Jiri Denemark <jdenemar>
AuthorDate: Mon Mar 13 12:32:02 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 17 11:50:48 2017 +0100

    qemu: Update CPU definition according to QEMU

    When starting a domain with custom guest CPU specification QEMU may add
    or remove some CPU features. There are several reasons for this, e.g.,
    QEMU/KVM does not support some requested features or the definition of
    the requested CPU model in libvirt's cpu_map.xml differs from the one
    QEMU is using. We can't really avoid this because CPU models are allowed
    to change with machine types and libvirt doesn't know (and probably
    doesn't even want to know) about such changes.

    Thus when we want to make sure guest ABI doesn't change when a domain
    gets migrated to another host, we need to update our live CPU definition
    according to the CPU QEMU created. Once updated, we will change CPU
    checking to VIR_CPU_CHECK_FULL to make sure the virtual CPU created
    after migration exactly matches the one on the source.

    https://bugzilla.redhat.com/show_bug.cgi?id=822148
    https://bugzilla.redhat.com/show_bug.cgi?id=824989

    Signed-off-by: Jiri Denemark <jdenemar>

commit 77ca2f6d8f6221f2991c95d74740c8607cd48baf
Refs: v3.1.0-193-g77ca2f6d8
Author:     Jiri Denemark <jdenemar>
AuthorDate: Tue Mar 14 15:05:02 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Fri Mar 17 11:50:48 2017 +0100

    qemu: Enforce guest CPU specification

    When guest CPU definition uses VIR_CPU_CHECK_FULL checks, we need to
    make sure QEMU does not add or remove any features.

    https://bugzilla.redhat.com/show_bug.cgi?id=822148
    https://bugzilla.redhat.com/show_bug.cgi?id=824989

    Signed-off-by: Jiri Denemark <jdenemar>

Comment 12 Jiri Denemark 2017-03-20 18:09:51 UTC

One more patch is needed to fix a possible crash:

commit c74207cb182f0af7ea626579f5be3be48fc9c2d4
Refs: v3.1.0-201-gc74207cb1
Author:     Jiri Denemark <jdenemar>
AuthorDate: Mon Mar 20 09:03:58 2017 +0100
Commit:     Jiri Denemark <jdenemar>
CommitDate: Mon Mar 20 09:03:58 2017 +0100

    qemu: Don't try to update undefined guest CPU

    Calling virCPUUpdateLive on a domain with no guest CPU configuration
    does not make sense. Especially when doing so would crash libvirtd.

    Signed-off-by: Jiri Denemark <jdenemar>

Comment 14 Luyao Huang 2017-06-20 09:52:50 UTC

Hit some problems during test migration, it will effect this bugs verification, see bug 1181899

Comment 15 Jiri Denemark 2017-06-21 14:27:31 UTC

The issue should be fixed in -14.

Comment 16 Luyao Huang 2017-06-22 05:24:08 UTC

Test with libvirt-3.2.0-14.el7.x86_64:
prepare source host support xsave and target host not support xsave

S1: custom + check full (in config)

1. Start a guest which config xml have:

  <cpu mode='custom' match='exact' check='full'>
    <model fallback='allow'>qemu64</model>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='xsave'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='lahf_lm'/>

2. check guest live xml and migrate xml:

live:
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>qemu64</model>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='xsave'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='lahf_lm'/>

migrate:
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>qemu64</model>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='xsave'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='lahf_lm'/>

3. migrate to target:


# virsh migrate r7 qemu+ssh://target/system --live
error: operation failed: guest CPU doesn't match specification: missing features: xsave


S2: custom + check none (in config):

1. Start a guest which config xml have:

  <cpu mode='custom' match='exact' check='none'>
    <model fallback='allow'>qemu64</model>
    <feature policy='require' name='xsave'/>


2. check guest live xml and migrate xml:

live:
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>qemu64</model>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='xsave'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='lahf_lm'/>

migrate:
  <cpu mode='custom' match='exact' check='none'>
    <model fallback='forbid'>qemu64</model>
    <feature policy='require' name='xsave'/>


3. migrate to target:


# virsh migrate r7 qemu+ssh://target/system --live
error: operation failed: guest CPU doesn't match specification: missing features: xsave


S3: custom + check partial

1. Start a guest which config xml have:

  <cpu mode='custom' match='exact' check='partial'>
    <model fallback='allow'>qemu64</model>
    <feature policy='require' name='xsave'/>

2. check guest live xml and migrate xml:

live:
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>qemu64</model>
    <feature policy='require' name='xsave'/>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='lahf_lm'/>


migrate:
  <cpu mode='custom' match='exact' check='partial'>
    <model fallback='forbid'>qemu64</model>
    <feature policy='require' name='xsave'/>


3. migrate to target:

# virsh migrate r7 qemu+ssh://target/system --live
error: operation failed: guest CPU doesn't match specification: missing features: xsave


Prepare another two host with different intel cpu to test host-model

S5: host-model + check full:

1. Start a guest which config xml have:

  <cpu mode='host-model' check='full'>
    <model fallback='allow'/>
    <feature policy='disable' name='xsaveopt'/>


2. check guest live xml and migrate xml:

live:
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>SandyBridge</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='pcid'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='arat'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='disable' name='xsave'/>
    <feature policy='disable' name='avx'/>
    <feature policy='disable' name='xsaveopt'/>



migrate:
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>SandyBridge</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='pcid'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='arat'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='disable' name='xsave'/>
    <feature policy='disable' name='avx'/>
    <feature policy='disable' name='xsaveopt'/>



3. migrate to target:

# virsh migrate r7 qemu+ssh://target/system --live
error: operation failed: guest CPU doesn't match specification: missing features: pdpe1gb


S6: host-model + check partial:

1. Start a guest which config xml have:

  <cpu mode='host-model' check='partial'>
    <model fallback='allow'/>


2. check guest live xml and migrate xml:

live:
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>SandyBridge</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='pcid'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='arat'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='disable' name='xsave'/>
    <feature policy='disable' name='avx'/>
    <feature policy='disable' name='xsaveopt'/>

migrate:

  <cpu mode='custom' match='exact' check='partial'>
    <model fallback='forbid'>SandyBridge</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='pcid'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='arat'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='disable' name='xsave'/>
    <feature policy='disable' name='avx'/>


3. migrate to target:


# virsh migrate r7 qemu+ssh://target/system --live
error: operation failed: guest CPU doesn't match specification: missing features: pdpe1gb

S7: host-model + check none:

1. Start a guest which config xml have:

  <cpu mode='host-model' check='none'>
    <model fallback='allow'/>

2. check guest live xml and migrate xml:

live:
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>SandyBridge</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='pcid'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='arat'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='disable' name='xsave'/>
    <feature policy='disable' name='avx'/>
    <feature policy='disable' name='xsaveopt'/>


migrate:

  <cpu mode='custom' match='exact' check='none'>
    <model fallback='forbid'>SandyBridge</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='pcid'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='arat'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='disable' name='xsave'/>
    <feature policy='disable' name='avx'/>


3. migrate to target:

# virsh migrate r7 qemu+ssh://target/system --live
error: operation failed: guest CPU doesn't match specification: missing features: pdpe1gb


And more test related to migration check bug 822148 comment 20

Comment 17 errata-xmlrpc 2017-08-01 17:06:41 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846

Comment 18 errata-xmlrpc 2017-08-01 23:48:45 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846

Comment 19 errata-xmlrpc 2017-08-02 01:25:04 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1846