Bug 1464804 - qemu guest fails to start with "unknown CPU feature invtsc" after upgrade to RHEL 7.4
qemu guest fails to start with "unknown CPU feature invtsc" after upgrade to ...
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt (Show other bugs)
7.4
Unspecified Unspecified
medium Severity medium
: rc
: ---
Assigned To: Libvirt Maintainers
Luyao Huang
virt
: Regression
: 1464802 (view as bug list)
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-25 18:44 EDT by Igor Netkachev
Modified: 2017-06-30 13:44 EDT (History)
10 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-06-30 13:44:26 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
the output of "rpm -V libvirt-client" (43 bytes, text/plain)
2017-06-29 11:20 EDT, pgozart
no flags Details
/usr/share/libvirt/cpu_map.xml (44.09 KB, text/plain)
2017-06-29 11:22 EDT, pgozart
no flags Details

  None (edit)
Description Igor Netkachev 2017-06-25 18:44:31 EDT
Description of problem:
qemu guest fails to start with "unknown CPU feature invtsc" after 7.3->7.4 upgrade


Version-Release number of selected component (if applicable):
RHEL 7.4

libvirt-3.2.0-9.el7.x86_64                                  Thu Jun 15 16:42:55 2017
libvirt-client-3.2.0-9.el7.x86_64                           Thu Jun 15 16:42:13 2017
libvirt-daemon-3.2.0-9.el7.x86_64                           Thu Jun 15 16:42:54 2017
libvirt-daemon-config-network-3.2.0-9.el7.x86_64            Thu Jun 15 16:42:54 2017
libvirt-daemon-config-nwfilter-3.2.0-9.el7.x86_64           Thu Jun 15 16:42:55 2017
libvirt-daemon-driver-interface-3.2.0-9.el7.x86_64          Thu Jun 15 16:42:54 2017
libvirt-daemon-driver-lxc-3.2.0-9.el7.x86_64                Thu Jun 15 16:42:54 2017
libvirt-daemon-driver-network-3.2.0-9.el7.x86_64            Thu Jun 15 16:42:54 2017
libvirt-daemon-driver-nodedev-3.2.0-9.el7.x86_64            Thu Jun 15 16:42:54 2017
libvirt-daemon-driver-nwfilter-3.2.0-9.el7.x86_64           Thu Jun 15 16:42:55 2017
libvirt-daemon-driver-qemu-3.2.0-9.el7.x86_64               Thu Jun 15 16:42:54 2017
libvirt-daemon-driver-secret-3.2.0-9.el7.x86_64             Thu Jun 15 16:42:54 2017
libvirt-daemon-driver-storage-3.2.0-9.el7.x86_64            Thu Jun 15 16:42:55 2017
libvirt-daemon-driver-storage-core-3.2.0-9.el7.x86_64       Thu Jun 15 16:42:54 2017
libvirt-daemon-driver-storage-disk-3.2.0-9.el7.x86_64       Thu Jun 15 16:42:54 2017
libvirt-daemon-driver-storage-gluster-3.2.0-9.el7.x86_64    Thu Jun 15 16:42:54 2017
libvirt-daemon-driver-storage-iscsi-3.2.0-9.el7.x86_64      Thu Jun 15 16:42:54 2017
libvirt-daemon-driver-storage-logical-3.2.0-9.el7.x86_64    Thu Jun 15 16:42:54 2017
libvirt-daemon-driver-storage-mpath-3.2.0-9.el7.x86_64      Thu Jun 15 16:42:55 2017
libvirt-daemon-driver-storage-rbd-3.2.0-9.el7.x86_64        Thu Jun 15 16:42:55 2017
libvirt-daemon-driver-storage-scsi-3.2.0-9.el7.x86_64       Thu Jun 15 16:42:54 2017
libvirt-daemon-kvm-3.2.0-9.el7.x86_64                       Thu Jun 15 16:43:20 2017
libvirt-gconfig-1.0.0-1.el7.x86_64                          Mon Jun 12 07:42:43 2017
libvirt-glib-1.0.0-1.el7.x86_64                             Mon Jun 12 07:44:05 2017
libvirt-gobject-1.0.0-1.el7.x86_64                          Mon Jun 12 07:44:05 2017
libvirt-libs-3.2.0-9.el7.x86_64                             Thu Jun 15 16:42:13 2017
libvirt-python-3.2.0-2.el7.x86_64                           Mon Jun 12 07:44:06 2017
qemu-guest-agent-2.8.0-2.el7.x86_64                         Mon Jun 12 07:48:30 2017
qemu-img-1.5.3-140.el7.x86_64                               Thu Jun 15 16:42:12 2017
qemu-kvm-1.5.3-140.el7.x86_64                               Thu Jun 15 16:42:42 2017
qemu-kvm-common-1.5.3-140.el7.x86_64                        Thu Jun 15 16:42:42 2017
qemu-kvm-tools-1.5.3-140.el7.x86_64                         Thu Jun 15 16:44:39 2017
qemu-kvm-tpm-2.6.0-28.el7_3.9.2.x86_64                      Wed Apr 26 12:06:57 2017


How reproducible:
100%

Steps to Reproduce:
1. Upgrade OS on Lenovo W540 T470 hypervisor to 7.4
2. Try to boot existing VM

Actual results:
error: Failed to start domain Virtual_Client_RHEL_6-KVM
error: internal error: unknown CPU feature invtsc


Expected results:
Guest boots without any issues

Additional info:
Downgrade libvirt to below versions makes guest boot again.
  libvirt.x86_64 0:3.2.0-6.el7                                                                
  libvirt-client.x86_64 0:3.2.0-6.el7                                                         
  libvirt-daemon.x86_64 0:3.2.0-6.el7                                                         
  libvirt-daemon-config-network.x86_64 0:3.2.0-6.el7                                          
  libvirt-daemon-config-nwfilter.x86_64 0:3.2.0-6.el7                                         
  libvirt-daemon-driver-interface.x86_64 0:3.2.0-6.el7                                        
  libvirt-daemon-driver-lxc.x86_64 0:3.2.0-6.el7                                              
  libvirt-daemon-driver-network.x86_64 0:3.2.0-6.el7                                          
  libvirt-daemon-driver-nodedev.x86_64 0:3.2.0-6.el7                                          
  libvirt-daemon-driver-nwfilter.x86_64 0:3.2.0-6.el7                                         
  libvirt-daemon-driver-qemu.x86_64 0:3.2.0-6.el7                                             
  libvirt-daemon-driver-secret.x86_64 0:3.2.0-6.el7                                           
  libvirt-daemon-driver-storage.x86_64 0:3.2.0-6.el7                                          
  libvirt-daemon-driver-storage-core.x86_64 0:3.2.0-6.el7                                     
  libvirt-daemon-driver-storage-disk.x86_64 0:3.2.0-6.el7                                     
  libvirt-daemon-driver-storage-gluster.x86_64 0:3.2.0-6.el7                                  
  libvirt-daemon-driver-storage-iscsi.x86_64 0:3.2.0-6.el7                                    
  libvirt-daemon-driver-storage-logical.x86_64 0:3.2.0-6.el7                                  
  libvirt-daemon-driver-storage-mpath.x86_64 0:3.2.0-6.el7                                    
  libvirt-daemon-driver-storage-rbd.x86_64 0:3.2.0-6.el7                                      
  libvirt-daemon-driver-storage-scsi.x86_64 0:3.2.0-6.el7                                     
  libvirt-daemon-kvm.x86_64 0:3.2.0-6.el7                                                     
  libvirt-gconfig.x86_64 0:0.2.3-1.el7                                                        
  libvirt-glib.x86_64 0:0.2.3-1.el7                                                           
  libvirt-gobject.x86_64 0:0.2.3-1.el7                                                        
  libvirt-libs.x86_64 0:3.2.0-6.el7                                                           
  libvirt-python.x86_64 0:2.0.0-2.el7

--
Kind Regards,
Igor Netkachev
Technical Support Engineer
Red Hat Global Support Services
Comment 2 Peter Krempa 2017-06-26 02:44:07 EDT
*** Bug 1464802 has been marked as a duplicate of this bug. ***
Comment 3 pgozart 2017-06-27 17:01:07 EDT
Customer provided the following additional information:

-------------

This occurs regardless of the guest OS.  Grant saw this with RHEL KVMs.  I see the same error with Win7 or Win10 KVMs (old or new).

I created a new VM:
1. Local install media
2. Use ISO image (Windows 10 ISO)
3. Check "Automatically detect operating system based on install media
4. Take defaults on all remaining panels and click Finish 
Unable to complete install: 'internal error: unknown CPU feature invtsc'

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 88, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/create.py", line 2288, in _do_async_install
    guest.start_install(meter=meter)
  File "/usr/share/virt-manager/virtinst/guest.py", line 497, in start_install
    doboot, transient)
  File "/usr/share/virt-manager/virtinst/guest.py", line 433, in _create_guest
    domain = self.conn.createXML(install_xml or final_xml, 0)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3567, in createXML
    if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
libvirtError: internal error: unknown CPU feature invtsc

Host OS is RHEL 7.4 snapshot 4.
[user@host ~]$ uname -r
3.10.0-681.el7.x86_64

libvirt-3.2.0-10.el7.x86_64
libvirt-client-3.2.0-10.el7.x86_64
libvirt-daemon-3.2.0-10.el7.x86_64
libvirt-daemon-config-network-3.2.0-10.el7.x86_64
libvirt-daemon-config-nwfilter-3.2.0-10.el7.x86_64
libvirt-daemon-driver-interface-3.2.0-10.el7.x86_64
libvirt-daemon-driver-lxc-3.2.0-10.el7.x86_64
libvirt-daemon-driver-network-3.2.0-10.el7.x86_64
libvirt-daemon-driver-nodedev-3.2.0-10.el7.x86_64
libvirt-daemon-driver-nwfilter-3.2.0-10.el7.x86_64
libvirt-daemon-driver-qemu-3.2.0-10.el7.x86_64
libvirt-daemon-driver-secret-3.2.0-10.el7.x86_64
libvirt-daemon-driver-storage-3.2.0-10.el7.x86_64
libvirt-daemon-driver-storage-core-3.2.0-10.el7.x86_64
libvirt-daemon-driver-storage-disk-3.2.0-10.el7.x86_64
libvirt-daemon-driver-storage-gluster-3.2.0-10.el7.x86_64
libvirt-daemon-driver-storage-iscsi-3.2.0-10.el7.x86_64
libvirt-daemon-driver-storage-logical-3.2.0-10.el7.x86_64
libvirt-daemon-driver-storage-mpath-3.2.0-10.el7.x86_64
libvirt-daemon-driver-storage-rbd-3.2.0-10.el7.x86_64
libvirt-daemon-driver-storage-scsi-3.2.0-10.el7.x86_64
libvirt-daemon-kvm-3.2.0-10.el7.x86_64
libvirt-glib-1.0.0-1.el7.x86_64
libvirt-libs-3.2.0-10.el7.x86_64
libvirt-python-3.2.0-3.el7.x86_64
Comment 4 Jiri Denemark 2017-06-28 11:39:50 EDT
This is really strange since the error message suggests libvirt can't find the definition of the invtsc feature in cpu_map.xml. I tried installing snap 4 locally and invtsc was correctly recognized.

So to be able to make any progress here, could you please enable debug logs for libvirtd by following http://wiki.libvirt.org/page/DebugLogs and provide the following data:

- the output of "rpm -V libvirt-client"
- /usr/share/libvirt/cpu_map.xml
- the output of "virsh capabilities"
- the output of "virsh domcapabilities"
- the domain XML which fails to start
- /var/log/libvirt/libvirtd.log
- /var/log/libvirt/qemu/$VMNAME.log
Comment 5 pgozart 2017-06-29 11:20 EDT
Created attachment 1292903 [details]
the output of "rpm -V libvirt-client"
Comment 7 pgozart 2017-06-29 11:22 EDT
Created attachment 1292905 [details]
/usr/share/libvirt/cpu_map.xml
Comment 12 pgozart 2017-06-29 11:29:21 EDT
(In reply to Jiri Denemark from comment #4)
> This is really strange since the error message suggests libvirt can't find
> the definition of the invtsc feature in cpu_map.xml. I tried installing snap
> 4 locally and invtsc was correctly recognized.
> 
> So to be able to make any progress here, could you please enable debug logs
> for libvirtd by following http://wiki.libvirt.org/page/DebugLogs and provide
> the following data:
> 
> - the output of "rpm -V libvirt-client"
> - /usr/share/libvirt/cpu_map.xml
> - the output of "virsh capabilities"
> - the output of "virsh domcapabilities"
> - the domain XML which fails to start
> - /var/log/libvirt/libvirtd.log
> - /var/log/libvirt/qemu/$VMNAME.log

Attachments added that provide the requested info as well as a video screencast of the customer replicating the problem.  -Paul
Comment 13 Jiri Denemark 2017-06-30 04:24:37 EDT
Thanks for all the data. Even though rpm -V doesn't show cpu_map.xml as modified in any way since it was installed, the file differs from the expected content:

diff --git 1/src/cpu/cpu_map.xml 2/cpu_map.xml
index 29b5b596f4..f9a4da3ea6 100644
--- 1/src/cpu/cpu_map.xml
+++ 2/cpu_map.xml
@@ -406,11 +406,6 @@
       <cpuid eax_in='0x80000001' ecx='0x01000000'/>
     </feature>
 
-    <!-- Advanced Power Management edx features -->
-    <feature name='invtsc' migratable='no'>
-      <cpuid eax_in='0x80000007' edx='0x00000100'/>
-    </feature>
-
     <!-- models -->
     <model name='486'>
       <feature name='fpu'/>

This would indeed explain why libvirt is complaining it doesn't know invtsc feature. And the file must have been correct at the time libvirt was built otherwise sever tens of tests would fail. So it must have been later somehow.
Comment 14 Jiri Denemark 2017-06-30 04:37:50 EDT
Oh, the file is now in libvirt-libs subpackage. I checked all packages from -6 to -14 and cpu_map.xml is correct in all of them. So could you please provide the output of "rpm -V libvirt-libs"?
Comment 16 Jaroslav Suchanek 2017-06-30 13:44:26 EDT
Cust. case closed, closing this bug too.

Note You need to log in before you can comment on or make changes to this bug.