1508549 – Post libvirt upgrade to 3.2.0-14, migration fails with -- "can't apply global Haswell-noTSX-x86_64-cpu.cmt=off: Property '.cmt' not found" [rhel-7.4.z]

RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.

Bug 1508549 - Post libvirt upgrade to 3.2.0-14, migration fails with -- "can't apply global Haswell-noTSX-x86_64-cpu.cmt=off: Property '.cmt' not found" [rhel-7.4.z]

Summary: Post libvirt upgrade to 3.2.0-14, migration fails with -- "can't apply global...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat Enterprise Linux 7
Classification:	Red Hat
Component:	libvirt
Sub Component:
Version:	7.2
Hardware:	x86_64
OS:	Linux
Priority:	urgent
Severity:	urgent
Target Milestone:	rc
Target Release:	---
Assignee:	Jiri Denemark
QA Contact:	zhe peng
Docs Contact:
URL:
Whiteboard:
Depends On:	1495171
Blocks:
TreeView+	depends on / blocked

Reported:	2017-11-01 16:14 UTC by Oneata Mircea Teodor
Modified:	2021-06-10 13:25 UTC (History)
CC List:	30 users (show)
Fixed In Version:	libvirt-3.2.0-14.el7_4.4
Doc Type:	Bug Fix
Doc Text:	Previously, the libvirt service in some cases added the "cmt" CPU feature incompatible with the QEMU emulator to KVM guest virtual machines with CPU set to "host-model". As a consequence, migrating or restoring these guests failed. With this update, libvirt no longer adds "cmt" to domain features and automatically removes "cmt" from guest configuration if present. As a result, the affected guests can be migrated and restored correctly.
Clone Of:	1495171
Environment:
Last Closed:	2017-11-30 16:06:34 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Red Hat Product Errata	RHBA-2017:3324	0	normal	SHIPPED_LIVE	libvirt bug fix update	2017-11-30 20:19:34 UTC

Description Oneata Mircea Teodor 2017-11-01 16:14:29 UTC

This bug has been copied from bug #1495171 and has been proposed to be backported to 7.4 z-stream (EUS).

Comment 6 zhe peng 2017-11-08 10:38:54 UTC

verify with build:
libvirt-3.2.0-14.el7_4.4.x86_64
qemu-kvm-rhev-2.9.0-16.el7_4.10.x86_64

step:
1. prepare a cmt host with build:
libvirt-1.2.17-13.el7_2.6.x86_64
qemu-kvm-rhev-2.3.0-31.el7_2.19.x86_64
2. create a guest with cpu model 'host-model'
....
 <cpu mode='host-model'>
    <model fallback='allow'>Haswell-noTSX</model>
    <vendor>Intel</vendor>
    <topology sockets='2' cores='1' threads='1'/>
  </cpu>
....
3. check cmd line
....
/usr/libexec/qemu-kvm -name rhel7.2 -S -machine pc-i440fx-rhel7.2.0,accel=kvm,usb=off -cpu Haswell-noTSX,+abm,+pdpe1gb,+rdrand,+f16c,+osxsave,+dca,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -m 4096 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 
....
4. upgrade libvirt to libvirt-3.2.0-14.el7_4.4.x86_64
5. check guest xml file
....
 <cpu mode='custom' match='exact' check='full'>
    <model fallback='allow'>Haswell-noTSX</model>
    <vendor>Intel</vendor>
    <topology sockets='2' cores='1' threads='1'/>
    <feature policy='require' name='vme'/>
    <feature policy='disable' name='ds'/>
    <feature policy='disable' name='acpi'/>
    <feature policy='require' name='ss'/>
    <feature policy='disable' name='ht'/>
    <feature policy='disable' name='tm'/>
    <feature policy='disable' name='pbe'/>
    <feature policy='disable' name='dtes64'/>
    <feature policy='disable' name='monitor'/>
    <feature policy='disable' name='ds_cpl'/>
    <feature policy='disable' name='vmx'/>
    <feature policy='disable' name='smx'/>
    <feature policy='disable' name='est'/>
    <feature policy='disable' name='tm2'/>
    <feature policy='disable' name='xtpr'/>
    <feature policy='disable' name='pdcm'/>
    <feature policy='disable' name='dca'/>
    <feature policy='disable' name='osxsave'/>
    <feature policy='require' name='f16c'/>
    <feature policy='require' name='rdrand'/>
    <feature policy='disable' name='arat'/>
    <feature policy='disable' name='tsc_adjust'/>
    <feature policy='require' name='xsaveopt'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='require' name='abm'/>
    <feature policy='require' name='hypervisor'/>
  </cpu>
....
6. migrate to another host(libvirt-3.2.0-14.el7_4.4.x86_64)
# virsh migrate --live rhel7.2 qemu+ssh://$target_host/system --verbose
Migration: [100 %]

we need do some regression testing for this, after that,i will change status
to verified.

Comment 7 Luyao Huang 2017-11-13 08:47:53 UTC

Hi Jirka,

I met a problem when test this bug with custom cpu model:

1. prepare a custom cpu model on a machine have cmt flags

# virsh dumpxml mig1

  <cpu mode='custom' match='minimum'>
    <model fallback='allow'>Haswell-noTSX</model>
    <feature policy='disable' name='invtsc'/>
  </cpu>

2. start guest:

# virsh start mig1
Domain mig1 started

3. update libvirt from libvirt-1.2.17-13.el7_2.6.x86_64 to libvirt-3.2.0-14.el7_4.4.x86_64

4. 

# virsh dumpxml mig1 --migratable > /tmp/mig1.xml

5. migrate to target host(libvirt-3.2.0-14.el7_4.4.x86_64)

# virsh migrate mig1 qemu+ssh://$target_host/system --live --xml /tmp/mig1.xml
error: internal error: qemu unexpectedly closed the monitor: 2017-11-13T08:32:42.966170Z qemu-kvm: -chardev pty,id=charserial0: char device redirected to /dev/pts/3 (label charserial0)
2017-11-13T08:32:42.968141Z qemu-kvm: can't apply global Haswell-noTSX-x86_64-cpu.cmt=on: Property '.cmt' not found


Could you please help to check if this is a problem worth to fix in the 7.4.z ?
Thanks a lot for your reply.

Comment 10 Jiri Denemark 2017-11-13 17:35:26 UTC

(In reply to Luyao Huang from comment #7)
> 3. update libvirt from libvirt-1.2.17-13.el7_2.6.x86_64 to
> libvirt-3.2.0-14.el7_4.4.x86_64
> 
> 4. # virsh dumpxml mig1 --migratable > /tmp/mig1.xml

Are you sure you did this after upgrading libvirt and when the new libvirt was
actually running? I can reproduce the issue only if I run this command
*before* upgrading libvirt, which is expected.


(In reply to Luyao Huang from comment #9)
> In function qemuDomainFixupCPUs() the variable ret was init as 0, and there
> is no other place to change this value to -1, so this function will always
> return 0.

Yeah, this is clearly a bug. Although pretty harmless, since it would just
cause qemuDomainFixupCPUs to report success in an unlikely OOM condition, but
some other function later will likely report the error anyway. And even if no
OOM was reported, the domain would just be left with the cmt feature and fixed
next time.

Comment 11 Luyao Huang 2017-11-14 02:04:00 UTC

(In reply to Jiri Denemark from comment #10)
> (In reply to Luyao Huang from comment #7)
> > 3. update libvirt from libvirt-1.2.17-13.el7_2.6.x86_64 to
> > libvirt-3.2.0-14.el7_4.4.x86_64
> > 
> > 4. # virsh dumpxml mig1 --migratable > /tmp/mig1.xml
> 
> Are you sure you did this after upgrading libvirt and when the new libvirt
> was
> actually running? I can reproduce the issue only if I run this command
> *before* upgrading libvirt, which is expected.
> 

Yes, i did this after update libvirt, also i found this problem is not related to the --xml options, migration still got failure after drop this option:

# virsh migrate mig1 qemu+ssh://$target_host/system --live
error: internal error: qemu unexpectedly closed the monitor: 2017-11-14T01:23:00.168327Z qemu-kvm: -chardev pty,id=charserial0: char device redirected to /dev/pts/3 (label charserial0)
2017-11-14T01:23:00.170506Z qemu-kvm: can't apply global Haswell-noTSX-x86_64-cpu.cmt=on: Property '.cmt' not found


> 
> (In reply to Luyao Huang from comment #9)
> > In function qemuDomainFixupCPUs() the variable ret was init as 0, and there
> > is no other place to change this value to -1, so this function will always
> > return 0.
> 
> Yeah, this is clearly a bug. Although pretty harmless, since it would just
> cause qemuDomainFixupCPUs to report success in an unlikely OOM condition, but
> some other function later will likely report the error anyway. And even if no
> OOM was reported, the domain would just be left with the cmt feature and
> fixed
> next time.

Okay, got it, thanks.

Comment 14 Jiri Denemark 2017-11-15 14:40:52 UTC

OK, I see the problem with <cpu mode='custom' match='minimum'> now. It should
already be fixed upstream and it would require backporting at least 10 more
patches. Since this kind of CPU configuration likely rare (it shared the same
issues we had with host-model, but we got no bug reports), I think it's better
to not backport the fixes.

However, there seems to be an additional small issue with <cpu mode='custom'
match='minimum'> when libvirt connects to a running domain started by old
libvirt. So please, file a separate bug for this issue.

Comment 16 Luyao Huang 2017-11-16 05:51:40 UTC

(In reply to Jiri Denemark from comment #14)
> OK, I see the problem with <cpu mode='custom' match='minimum'> now. It should
> already be fixed upstream and it would require backporting at least 10 more
> patches. Since this kind of CPU configuration likely rare (it shared the same
> issues we had with host-model, but we got no bug reports), I think it's
> better
> to not backport the fixes.
> 
> However, there seems to be an additional small issue with <cpu mode='custom'
> match='minimum'> when libvirt connects to a running domain started by old
> libvirt. So please, file a separate bug for this issue.

Okay, got it, i have retested this issue with libvirt-3.9.0-2.el7.x86_64, and didn't hit this issue.

Comment 18 zhe peng 2017-11-16 08:09:14 UTC

per comment 6,14,16
move this bug to verified.

Comment 21 errata-xmlrpc 2017-11-30 16:06:34 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:3324

Note You need to log in before you can comment on or make changes to this bug.

aglotov
anrussel
berrange
chhu
dasmith
dhill
dvd
dyuan
eglynn
jbryant
jdenemar
jherrman
jishao
kchamart
lhuang
mkalinin
pneedle
rbalakri
rbryant
rhodain
sbauza
sferdjao
sgordon
smykhail
srevivo
thomas.oulevey
toneata
vromanso
xuzhang
zpeng