RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1057251 - openstack-nova-compute service fails with - libvirtError: internal error: CPU feature `avx' specified more than once
Summary: openstack-nova-compute service fails with - libvirtError: internal error: CP...
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: libvirt
Version: 7.0
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Jiri Denemark
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On: 1049391
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-01-23 17:24 UTC by Kashyap Chamarthy
Modified: 2014-01-28 21:36 UTC (History)
21 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of: 1049391
Environment:
Last Closed: 2014-01-28 21:36:04 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1267191 0 None None None Never

Description Kashyap Chamarthy 2014-01-23 17:24:16 UTC
+++ This bug was initially created as a clone of Bug #1049391 +++

[Note, cloned only the 'Description', for all the attachments, further discussions, please refer the Fedora/Libvirt bug.]

Description of problem
----------------------

Restarting openstack-nova-compute serivce fails with:

  libvirtError: internal error: CPU feature `avx' specified more than once

Version
-------

    $ rpm -q openstack-nova libvirt qemu-system-x86
    openstack-nova-2014.1-0.4.b1.fc21.noarch
    libvirt-1.1.3.2-1.fc20.x86_64
    qemu-system-x86-1.6.1-3.fc20.x86_64


Test env
--------

A two node OpenStack RDO set-up configured manually on two Fedora 20
VMs:

  - Controller node: Nova, Keystone, Cinder, Glance, Neutron (using Open
    vSwitch plugin and GRE tunneling).

  - Compute node: Nova (nova-compute), Neutron (openvswitch-agent)


How reproducible: Consistently.


Steps to Reproduce
------------------

    $ systemctl restart openstack-nova-compute

Observe /var/log/nova/compute.log


Actual results 
--------------


$ tail -f /var/log/nova/compute.log
2014-01-07 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 187, in doit
2014-01-07 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup     result = proxy_call(self._autowrap, f, *args, **kwargs)
2014-01-07 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 147, in proxy_call
2014-01-07 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup     rv = execute(f,*args,**kwargs)
2014-01-07 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 76, in tworker
2014-01-07 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup     rv = meth(*args,**kwargs)
2014-01-07 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3622, in baselineCPU
2014-01-07 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup     if ret is None: raise libvirtError ('virConnectBaselineCPU() failed', conn=self)
2014-01-07 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup libvirtError: internal error: CPU feature `avx' specified more than once
2014-01-07 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup 


Expected results
----------------

Compute service should start successfully.

Additional info
---------------

Status of openstack-nova-compute service

$ systemctl status openstack-nova-compute
openstack-nova-compute.service - OpenStack Nova Compute Server
   Loaded: loaded (/usr/lib/systemd/system/openstack-nova-compute.service; enabled)
   Active: inactive (dead) since Tue 2014-01-07 07:00:07 EST; 13min ago
  Process: 1529 ExecStart=/usr/bin/nova-compute --logfile /var/log/nova/compute.log (code=exited, status=0/SUCCESS)
 Main PID: 1529 (code=exited, status=0/SUCCESS)

Jan 07 07:00:07 node2-compute nova-compute[1529]: 2014-01-07 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup   File "/usr/lib/pyth...in doit
Jan 07 07:00:07 node2-compute nova-compute[1529]: 2014-01-07 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup     result = proxy_ca...kwargs)
Jan 07 07:00:07 node2-compute nova-compute[1529]: 2014-01-07 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup   File "/usr/lib/pyth...xy_call
Jan 07 07:00:07 node2-compute nova-compute[1529]: 2014-01-07 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup     rv = execute(f,*a...kwargs)
Jan 07 07:00:07 node2-compute nova-compute[1529]: 2014-01-07 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup   File "/usr/lib/pyth...tworker
Jan 07 07:00:07 node2-compute nova-compute[1529]: 2014-01-07 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup     rv = meth(*args,**kwargs)
Jan 07 07:00:07 node2-compute nova-compute[1529]: 2014-01-07 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup   File "/usr/lib64/py...lineCPU
Jan 07 07:00:07 node2-compute nova-compute[1529]: 2014-01-07 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup     if ret is None: r...n=self)
Jan 07 07:00:07 node2-compute nova-compute[1529]: 2014-01-07 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup libvirtError: interna...an once
Jan 07 07:00:07 node2-compute nova-compute[1529]: 2014-01-07 07:00:07.200 1529 TRACE nova.openstack.common.threadgroup

--- Additional comment from Kashyap Chamarthy on 2014-01-07 09:21:17 EST ---

More contextual trace from compute.log:

[. . .]
2012-12-10 22:12:38.789 1429 TRACE nova.virt.libvirt.driver 
2012-12-10 22:12:39.319 1429 ERROR nova.openstack.common.threadgroup [-] internal error: CPU feature `avx' specified more than once
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/openstack/common/threadgroup.py", line 117, in wait
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     x.wait()
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/openstack/common/threadgroup.py", line 49, in wait
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     return self.thread.wait()
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 168, in wait
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     return self._exit_event.wait()
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 116, in wait
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     return hubs.get_hub().switch()
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 187, in switch
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     return self.greenlet.switch()
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     result = function(*args, **kwargs)
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/openstack/common/service.py", line 448, in run_service
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     service.start()
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/service.py", line 164, in start
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     self.manager.pre_start_hook()
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 822, in pre_start_hook
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     self.update_available_resource(nova.context.get_admin_context())
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4971, in update_available_resource
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     nodenames = set(self.driver.get_available_nodes())
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/virt/driver.py", line 980, in get_available_nodes
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     stats = self.get_host_stats(refresh=refresh)
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4569, in get_host_stats
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     return self.host_state.get_host_stats(refresh=refresh)
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 429, in host_state
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     self._host_state = HostState(self)
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4960, in __init__
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     self.update_status()
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4999, in update_status
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     self.driver.get_instance_capabilities()
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3702, in get_instance_capabilities
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     caps = self.get_host_capabilities()
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2742, in get_host_capabilities
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     libvirt.VIR_CONNECT_BASELINE_CPU_EXPAND_FEATURES)
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 187, in doit
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     result = proxy_call(self._autowrap, f, *args, **kwargs)
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 147, in proxy_call
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     rv = execute(f,*args,**kwargs)
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 76, in tworker
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     rv = meth(*args,**kwargs)
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3622, in baselineCPU
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup     if ret is None: raise libvirtError ('virConnectBaselineCPU() failed', conn=self)
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup libvirtError: internal error: CPU feature `avx' specified more than once
2012-12-10 22:12:39.319 1429 TRACE nova.openstack.common.threadgroup 
2012-12-10 22:12:58.904 1467 WARNING nova.virt.libvirt.driver [req-88723bba-e677-4956-9b70-1f7fe66dbff7 None None] Cannot update service status on host: node2-compute,due to an unexpected exception.
[. . .]

Comment 2 Jiri Denemark 2014-01-27 23:24:50 UTC
Patches sent upstream for review: https://www.redhat.com/archives/libvir-list/2014-January/msg01314.html

Comment 3 Jiri Denemark 2014-01-28 20:32:08 UTC
Fixed upstream by commits v1.2.1-78-g0e9373a..v1.2.1-81-g7e4dcf3:

commit 0e9373a5c04e5c69a95b6d1cae7cad8456332be7
Author: Jiri Denemark <jdenemar>
Date:   Tue Jan 28 00:00:44 2014 +0100

    tests: Better support for VIR_CONNECT_BASELINE_CPU_EXPAND_FEATURES
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1049391
    
    virConnectBaselineCPU test results are now stored in different files
    depending on VIR_CONNECT_BASELINE_CPU_EXPAND_FEATURES.

commit 802f157e8c7afaf4971b82705f67519939d2da0c
Author: Jiri Denemark <jdenemar>
Date:   Mon Jan 27 17:03:55 2014 +0100

    cpu: Fix VIR_CONNECT_BASELINE_CPU_EXPAND_FEATURES
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1049391
    
    VIR_CONNECT_BASELINE_CPU_EXPAND_FEATURES flag for virConnectBaselineCPU
    did not work if the resulting guest CPU would disable some features
    present in its base model. This patch makes sure we won't try to add
    such features twice.

commit 580ddf0d34ad61b26357b94783ff622d0e73cb2b
Author: Jiri Denemark <jdenemar>
Date:   Mon Jan 27 20:41:43 2014 +0100

    cpu: Try to use source CPU model in virConnectBaselineCPU
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1049391
    
    When all source CPU XMLs contain just a single CPU model (with a
    possibly varying set of additional feature elements),
    virConnectBaselineCPU will try to use this CPU model in the computed
    guest CPU. Thus, when used on just a single CPU (useful with
    VIR_CONNECT_BASELINE_CPU_EXPAND_FEATURES), the result will not use a
    different CPU model.
    
    If the computed CPU uses the source model, set fallback mode to 'forbid'
    to make sure the guest CPU will always be as close as possible to the
    source CPUs.

commit 7e4dcf3a47a4fea8534531cf0a4bee7bb85904be
Author: Jiri Denemark <jdenemar>
Date:   Mon Jan 27 21:53:51 2014 +0100

    tests: Add more tests for virConnectBaselineCPU
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1049391
    
    The new tests would fail in various ways without the two previous
    commits.

Comment 4 Jiri Denemark 2014-01-28 21:36:04 UTC
We don't even support VIR_CONNECT_BASELINE_CPU_EXPAND_FEATURES in 7.0, thus we can't have bugs in it :-)


Note You need to log in before you can comment on or make changes to this bug.