Bug 1056144

Summary: nova-compute service is not started after installation on Fedora 20
Product: [Community] RDO Reporter: David Xie <davidx>
Component: openstack-packstackAssignee: RHOS Maint <rhos-maint>
Status: CLOSED DUPLICATE QA Contact: Ami Jeain <ajeain>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: unspecifiedCC: aortega, berrange, chrisw, davidx, derekh, fvollero, kchamart, mestery, ndipanov, yeylon
Target Milestone: ---   
Target Release: Icehouse   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2014-01-27 04:32:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description David Xie 2014-01-21 15:06:35 UTC 
Description of problem:
nova-compute service is not started after installation on Fedora 20

Version-Release number of selected component (if applicable):


How reproducible:
Yes

Steps to Reproduce:
1. get latest packstack from github.com and use master branch
2. install http://repos.fedorapeople.org/repos/openstack/openstack-icehouse/fedora-20/rdo-release-icehouse-1.noarch.rpm
3. `packstack --allinone --os-heat-install=y`

Actual results:
[davidx@fedora ~(keystone_admin)]$ nova service-list
+------------------+--------+----------+---------+-------+----------------------------+-----------------+
| Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+------------------+--------+----------+---------+-------+----------------------------+-----------------+
| nova-consoleauth | fedora | internal | enabled | up | 2014-01-21T14:24:18.000000 | None |
| nova-scheduler | fedora | internal | enabled | up | 2014-01-21T14:24:19.000000 | None |
| nova-conductor | fedora | internal | enabled | up | 2014-01-21T14:24:18.000000 | None |
| nova-compute | fedora | nova | enabled | down | None | None |
| nova-cert | fedora | internal | enabled | up | 2014-01-21T14:24:18.000000 | None |
+------------------+--------+----------+---------+-------+----------------------------+-----------------+

Expected results:
nova-compute is started

Additional info:
cat /var/log/nova/compute.log:

2014-01-21 22:22:40.755 2042 ERROR nova.openstack.common.threadgroup [-] internal error: CPU feature `tsc-deadline' specified more than once
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/openstack/common/threadgroup.py", line 117, in wait
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup x.wait()
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/openstack/common/threadgroup.py", line 49, in wait
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup return self.thread.wait()
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 168, in wait
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup return self._exit_event.wait()
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 116, in wait
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup return hubs.get_hub().switch()
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 187, in switch
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup return self.greenlet.switch()
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup result = function(*args, **kwargs)
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/openstack/common/service.py", line 448, in run_service
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup service.start()
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/service.py", line 164, in start
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup self.manager.pre_start_hook()
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 822, in pre_start_hook
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup self.update_available_resource(nova.context.get_admin_context())
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 4971, in update_available_resource
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup nodenames = set(self.driver.get_available_nodes())
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/virt/driver.py", line 980, in get_available_nodes
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup stats = self.get_host_stats(refresh=refresh)
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4569, in get_host_stats
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup return self.host_state.get_host_stats(refresh=refresh)
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 429, in host_state
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup self._host_state = HostState(self)
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4960, in __init__
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup self.update_status()
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4999, in update_status
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup self.driver.get_instance_capabilities()
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3702, in get_instance_capabilities
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup caps = self.get_host_capabilities()
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2742, in get_host_capabilities
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup libvirt.VIR_CONNECT_BASELINE_CPU_EXPAND_FEATURES)
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 187, in doit
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup result = proxy_call(self._autowrap, f, *args, **kwargs)
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 147, in proxy_call
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup rv = execute(f,*args,**kwargs)
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 76, in tworker
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup rv = meth(*args,**kwargs)
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3622, in baselineCPU
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup if ret is None: raise libvirtError ('virConnectBaselineCPU() failed', conn=self)
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup libvirtError: internal error: CPU feature `tsc-deadline' specified more than once
2014-01-21 22:22:40.755 2042 TRACE nova.openstack.common.threadgroup
[davidx@fedora ~(keystone_admin)]$
Comment 2 Francesco Vollero 2014-01-21 18:47:29 UTC 
We have just recently created the stable/havana branch.
Can you try with havana instead of icehouse ?
Comment 3 Kyle Mestery 2014-01-23 14:20:57 UTC 
I see a similar problem happening in upstream Icehouse with Fedora 20 as well. This works just fine with Fedora 19, so I suspect something package-wise must be different:

2014-01-22 17:57:29.264 25388 DEBUG nova.virt.libvirt.config [-] Generated XML ('<cpu>\n  <arch>x86_64</arch>\n  <model>Westmere</model>\n  <vendor>Intel</vendor>\n  <topology sockets="2" cores="1" threads="1"/>\n  <feature name="rdtscp"/>\n  <feature name="hypervisor"/>\n  <feature name="avx"/>\n  <feature name="osxsave"/>\n  <feature name="xsave"/>\n  <feature name="x2apic"/>\n  <feature name="pcid"/>\n  <feature name="vmx"/>\n  <feature name="pclmuldq"/>\n  <feature name="ss"/>\n  <feature name="ds"/>\n  <feature name="vme"/>\n</cpu>\n',)  to_xml /opt/stack/nova/nova/virt/libvirt/config.py:73
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 346, in fire_timers
    timer()
  File "/usr/lib/python2.7/site-packages/eventlet/hubs/timer.py", line 56, in __call__
    cb(*args, **kw)
  File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 163, in _do_send
    waiter.switch(result)
  File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main
    result = function(*args, **kwargs)
  File "/opt/stack/nova/nova/openstack/common/service.py", line 480, in run_service
    service.start()
  File "/opt/stack/nova/nova/service.py", line 182, in start
    self.manager.pre_start_hook()
  File "/opt/stack/nova/nova/compute/manager.py", line 815, in pre_start_hook
    self.update_available_resource(nova.context.get_admin_context())
  File "/opt/stack/nova/nova/compute/manager.py", line 5012, in update_available_resource
    nodenames = set(self.driver.get_available_nodes())
  File "/opt/stack/nova/nova/virt/driver.py", line 1058, in get_available_nodes
    stats = self.get_host_stats(refresh=refresh)
  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4573, in get_host_stats
    return self.host_state.get_host_stats(refresh=refresh)
  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 424, in host_state
    self._host_state = HostState(self)
  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4964, in __init__
    self.update_status()
  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5003, in update_status
    self.driver.get_instance_capabilities()
  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3697, in get_instance_capabilities
    caps = self.get_host_capabilities()
  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2765, in get_host_capabilities
    libvirt.VIR_CONNECT_BASELINE_CPU_EXPAND_FEATURES)
  File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 187, in doit
    result = proxy_call(self._autowrap, f, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 147, in proxy_call
    rv = execute(f,*args,**kwargs)
  File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 76, in tworker
    rv = meth(*args,**kwargs)
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3622, in baselineCPU
    if ret is None: raise libvirtError ('virConnectBaselineCPU() failed', conn=self)
libvirtError: internal error: CPU feature `tsc-deadline' specified more than once
2014-01-22 17:57:29.280 25388 ERROR nova.openstack.common.threadgroup [-] internal error: CPU feature `tsc-deadline' specified more than once
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup Traceback (most recent call last):
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/openstack/common/threadgroup.py", line 117, in wait
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup     x.wait()
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/openstack/common/threadgroup.py", line 49, in wait
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup     return self.thread.wait()
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 168, in wait
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup     return self._exit_event.wait()
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 116, in wait
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup     return hubs.get_hub().switch()
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 187, in switch
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup     return self.greenlet.switch()
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup     result = function(*args, **kwargs)
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/openstack/common/service.py", line 480, in run_service
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup     service.start()
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/service.py", line 182, in start
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup     self.manager.pre_start_hook()
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/compute/manager.py", line 815, in pre_start_hook
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup     self.update_available_resource(nova.context.get_admin_context())
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/compute/manager.py", line 5012, in update_available_resource
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup     nodenames = set(self.driver.get_available_nodes())
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/virt/driver.py", line 1058, in get_available_nodes
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup     stats = self.get_host_stats(refresh=refresh)
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4573, in get_host_stats
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup     return self.host_state.get_host_stats(refresh=refresh)
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 424, in host_state
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup     self._host_state = HostState(self)
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 4964, in __init__
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup     self.update_status()
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5003, in update_status
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup     self.driver.get_instance_capabilities()
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3697, in get_instance_capabilities
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup     caps = self.get_host_capabilities()
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup   File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2765, in get_host_capabilities
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup     libvirt.VIR_CONNECT_BASELINE_CPU_EXPAND_FEATURES)
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 187, in doit
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup     result = proxy_call(self._autowrap, f, *args, **kwargs)
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 147, in proxy_call
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup     rv = execute(f,*args,**kwargs)
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 76, in tworker
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup     rv = meth(*args,**kwargs)
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3622, in baselineCPU
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup     if ret is None: raise libvirtError ('virConnectBaselineCPU() failed', conn=self)
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup libvirtError: internal error: CPU feature `tsc-deadline' specified more than once
2014-01-22 17:57:29.280 25388 TRACE nova.openstack.common.threadgroup 
[kmestery@fedora-mac tempest]$ [kmestery@fedora-mac devstack]$
Comment 4 David Xie 2014-01-23 15:37:14 UTC 
I did try Havana, but I got another issue about nova-network and logged it on launchpad: https://bugs.launchpad.net/packstack/+bug/1271096
Should I log that network issue on bugzilla?
Comment 5 Chris Wright 2014-01-24 22:01:42 UTC 
Dan, do you have any idea?  This is blocking our OpenDaylight work on F20.
Comment 6 Francesco Vollero 2014-01-25 12:07:44 UTC 
It seems to be a libvirtd bug as mentioned in this bug over here too https://bugs.launchpad.net/nova/+bug/1267191 and here bug 1049391 even if the cpu flag is different in their case.
Comment 7 Kashyap Chamarthy 2014-01-27 04:27:41 UTC 
This is a duplicate of the below bug, where you could see the root-cause identified:

  https://bugzilla.redhat.com/show_bug.cgi?id=1049391#c11

And, related upstream bug:

  https://bugs.launchpad.net/nova/+bug/1267191
Comment 8 Kashyap Chamarthy 2014-01-27 04:32:12 UTC 

*** This bug has been marked as a duplicate of bug 1049391 ***