1511183 – Instance status is 'ERROR' for all instances

Bug 1511183 - Instance status is 'ERROR' for all instances

Summary: Instance status is 'ERROR' for all instances

Keywords:
Status:	CLOSED WORKSFORME
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-nova
Sub Component:
Version:	10.0 (Newton)
Hardware:	x86_64
OS:	Linux
Priority:	high
Severity:	high
Target Milestone:	---
Target Release:	---
Assignee:	Eoghan Glynn
QA Contact:	Joe H. Rahme
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:	1481685
TreeView+	depends on / blocked

Reported:	2017-11-08 20:37 UTC by Rachana Patel
Modified:	2019-09-09 14:07 UTC (History)
CC List:	11 users (show)
Fixed In Version:
Doc Type:	If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:	2017-11-15 06:08:32 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Description Rachana Patel 2017-11-08 20:37:01 UTC

Description of problem:
=======================

[stack@refarch-r630-01 ~]$ openstack server list
+--------------------------------------+-------+--------+----------+------------+
| ID                                   | Name  | Status | Networks | Image Name |
+--------------------------------------+-------+--------+----------+------------+
| c49972ee-f585-4481-ad28-7a002590fcac | inst2 | ERROR  |          | rhel7      |
| 11dc1567-97d2-457c-9232-70a2e4dce519 | inst1 | ERROR  |          | rhel7      |
+--------------------------------------+-------+--------+----------+------------+

All instances have 'error' status


Version-Release number of selected component (if applicable):
=============================================================
OSP 10 
Ceph 2.x

How reproducible:
==================
always


Steps to Reproduce:
===================
1. subscribe to OSP10 and ceph 2.x
2. Deploy Hyper converged overcloud where MONs are collocated with controller and OSDs are collacted with compute (3 Nodes for controller and 3 nodes for OSD-compute)
3. create instances 

Actual results:
==============
Instances have 'Error' status


Additional info:
================

[root@overcloud-osd-compute-0 ~]# grep 'c49972ee' /var/log/nova/nova-compute.log 
2017-11-08 19:47:19.947 63034 INFO nova.compute.claims [req-5c2ae3a1-08f2-46ee-8624-82245dc4773e 0393244be57c4359b3522cd0b87c825d 4f256e5fbc294a0cbd50f793c5838a32 - - -] [instance: c49972ee-f585-4481-ad28-7a002590fcac] Attempting claim: memory 4096 MB, disk 40 GB, vcpus 2 CPU
2017-11-08 19:47:19.948 63034 INFO nova.compute.claims [req-5c2ae3a1-08f2-46ee-8624-82245dc4773e 0393244be57c4359b3522cd0b87c825d 4f256e5fbc294a0cbd50f793c5838a32 - - -] [instance: c49972ee-f585-4481-ad28-7a002590fcac] Total memory: 65442 MB, used: 31096.00 MB
2017-11-08 19:47:19.948 63034 INFO nova.compute.claims [req-5c2ae3a1-08f2-46ee-8624-82245dc4773e 0393244be57c4359b3522cd0b87c825d 4f256e5fbc294a0cbd50f793c5838a32 - - -] [instance: c49972ee-f585-4481-ad28-7a002590fcac] memory limit: 65442.00 MB, free: 34346.00 MB
2017-11-08 19:47:19.948 63034 INFO nova.compute.claims [req-5c2ae3a1-08f2-46ee-8624-82245dc4773e 0393244be57c4359b3522cd0b87c825d 4f256e5fbc294a0cbd50f793c5838a32 - - -] [instance: c49972ee-f585-4481-ad28-7a002590fcac] Total disk: 89368 GB, used: 40.00 GB
2017-11-08 19:47:19.948 63034 INFO nova.compute.claims [req-5c2ae3a1-08f2-46ee-8624-82245dc4773e 0393244be57c4359b3522cd0b87c825d 4f256e5fbc294a0cbd50f793c5838a32 - - -] [instance: c49972ee-f585-4481-ad28-7a002590fcac] disk limit: 89368.00 GB, free: 89328.00 GB
2017-11-08 19:47:19.949 63034 INFO nova.compute.claims [req-5c2ae3a1-08f2-46ee-8624-82245dc4773e 0393244be57c4359b3522cd0b87c825d 4f256e5fbc294a0cbd50f793c5838a32 - - -] [instance: c49972ee-f585-4481-ad28-7a002590fcac] Total vcpu: 32 VCPU, used: 2.00 VCPU
2017-11-08 19:47:19.949 63034 INFO nova.compute.claims [req-5c2ae3a1-08f2-46ee-8624-82245dc4773e 0393244be57c4359b3522cd0b87c825d 4f256e5fbc294a0cbd50f793c5838a32 - - -] [instance: c49972ee-f585-4481-ad28-7a002590fcac] vcpu limit not specified, defaulting to unlimited
2017-11-08 19:47:19.950 63034 INFO nova.compute.claims [req-5c2ae3a1-08f2-46ee-8624-82245dc4773e 0393244be57c4359b3522cd0b87c825d 4f256e5fbc294a0cbd50f793c5838a32 - - -] [instance: c49972ee-f585-4481-ad28-7a002590fcac] Claim successful
2017-11-08 19:47:20.208 63034 INFO nova.network.neutronv2.api [req-5c2ae3a1-08f2-46ee-8624-82245dc4773e 0393244be57c4359b3522cd0b87c825d 4f256e5fbc294a0cbd50f793c5838a32 - - -] [instance: c49972ee-f585-4481-ad28-7a002590fcac] Updating port 06df74f7-a600-4500-87ff-0cb0a41dccc6 with attributes {'binding:host_id': u'overcloud-osd-compute-0.localdomain'}
2017-11-08 19:47:21.205 63034 WARNING nova.virt.libvirt.driver [req-5c2ae3a1-08f2-46ee-8624-82245dc4773e 0393244be57c4359b3522cd0b87c825d 4f256e5fbc294a0cbd50f793c5838a32 - - -] [instance: c49972ee-f585-4481-ad28-7a002590fcac] Ignoring supplied device name: /dev/vdb. Libvirt can't honour user-supplied dev names
2017-11-08 19:47:21.528 63034 INFO nova.virt.libvirt.driver [req-5c2ae3a1-08f2-46ee-8624-82245dc4773e 0393244be57c4359b3522cd0b87c825d 4f256e5fbc294a0cbd50f793c5838a32 - - -] [instance: c49972ee-f585-4481-ad28-7a002590fcac] Creating image
  <uuid>c49972ee-f585-4481-ad28-7a002590fcac</uuid>
      <entry name="uuid">c49972ee-f585-4481-ad28-7a002590fcac</entry>
      <source protocol="rbd" name="vms/c49972ee-f585-4481-ad28-7a002590fcac_disk">
      <source protocol="rbd" name="vms/c49972ee-f585-4481-ad28-7a002590fcac_disk.eph0">
      <source path="/var/lib/nova/instances/c49972ee-f585-4481-ad28-7a002590fcac/console.log"/>
2017-11-08 19:47:37.481 63034 ERROR nova.virt.libvirt.driver [req-5c2ae3a1-08f2-46ee-8624-82245dc4773e 0393244be57c4359b3522cd0b87c825d 4f256e5fbc294a0cbd50f793c5838a32 - - -] [instance: c49972ee-f585-4481-ad28-7a002590fcac] Failed to start libvirt guest
2017-11-08 19:47:37.703 63034 INFO nova.virt.libvirt.driver [req-5c2ae3a1-08f2-46ee-8624-82245dc4773e 0393244be57c4359b3522cd0b87c825d 4f256e5fbc294a0cbd50f793c5838a32 - - -] [instance: c49972ee-f585-4481-ad28-7a002590fcac] Deleting instance files /var/lib/nova/instances/c49972ee-f585-4481-ad28-7a002590fcac_del
2017-11-08 19:47:37.703 63034 INFO nova.virt.libvirt.driver [req-5c2ae3a1-08f2-46ee-8624-82245dc4773e 0393244be57c4359b3522cd0b87c825d 4f256e5fbc294a0cbd50f793c5838a32 - - -] [instance: c49972ee-f585-4481-ad28-7a002590fcac] Deletion of /var/lib/nova/instances/c49972ee-f585-4481-ad28-7a002590fcac_del complete
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [req-5c2ae3a1-08f2-46ee-8624-82245dc4773e 0393244be57c4359b3522cd0b87c825d 4f256e5fbc294a0cbd50f793c5838a32 - - -] [instance: c49972ee-f585-4481-ad28-7a002590fcac] Instance failed to spawn
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac] Traceback (most recent call last):
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2083, in _build_resources
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]     yield resources
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1924, in _build_and_run_instance
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]     block_device_info=block_device_info)
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2661, in spawn
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]     destroy_disks_on_failure=True)
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4969, in _create_domain_and_network
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]     destroy_disks_on_failure)
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]     self.force_reraise()
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]     six.reraise(self.type_, self.value, self.tb)
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4941, in _create_domain_and_network
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]     post_xml_callback=post_xml_callback)
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4852, in _create_domain
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]     guest = libvirt_guest.Guest.create(xml, self._host)
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 129, in create
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]     encodeutils.safe_decode(xml))
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]     self.force_reraise()
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]     six.reraise(self.type_, self.value, self.tb)
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 125, in create
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]     domain = host.write_instance_config(xml)
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/host.py", line 906, in write_instance_config
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]     return self.get_connection().defineXML(xml)
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 186, in doit
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 144, in proxy_call
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]     rv = execute(f, *args, **kwargs)
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 125, in execute
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]     six.reraise(c, e, tb)
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 83, in tworker
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]     rv = meth(*args, **kwargs)
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3585, in defineXML
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac]     if ret is None:raise libvirtError('virDomainDefineXML() failed', conn=self)
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac] libvirtError: invalid argument: could not find capabilities for domaintype=kvm 
2017-11-08 19:47:37.837 63034 ERROR nova.compute.manager [instance: c49972ee-f585-4481-ad28-7a002590fcac] 
2017-11-08 19:47:37.838 63034 INFO nova.compute.manager [req-5c2ae3a1-08f2-46ee-8624-82245dc4773e 0393244be57c4359b3522cd0b87c825d 4f256e5fbc294a0cbd50f793c5838a32 - - -] [instance: c49972ee-f585-4481-ad28-7a002590fcac] Terminating instance
2017-11-08 19:47:37.843 63034 INFO nova.virt.libvirt.driver [-] [instance: c49972ee-f585-4481-ad28-7a002590fcac] During wait destroy, instance disappeared.
2017-11-08 19:47:37.902 63034 INFO nova.virt.libvirt.driver [req-5c2ae3a1-08f2-46ee-8624-82245dc4773e 0393244be57c4359b3522cd0b87c825d 4f256e5fbc294a0cbd50f793c5838a32 - - -] [instance: c49972ee-f585-4481-ad28-7a002590fcac] Deletion of /var/lib/nova/instances/c49972ee-f585-4481-ad28-7a002590fcac_del complete
2017-11-08 19:47:38.053 63034 INFO nova.compute.manager [req-5c2ae3a1-08f2-46ee-8624-82245dc4773e 0393244be57c4359b3522cd0b87c825d 4f256e5fbc294a0cbd50f793c5838a32 - - -] [instance: c49972ee-f585-4481-ad28-7a002590fcac] Took 0.21 seconds to destroy the instance on the hypervisor.
[root@overcloud-osd-compute-0 ~]# cat nova.conf | grep "virt_t"
cat: nova.conf: No such file or directory
[root@overcloud-osd-compute-0 ~]# cat /etc/nova/nova.conf | grep "virt_t"
virt_type=kvm
[root@overcloud-osd-compute-0 ~]# lsmod | grep kvm
kvm                   566340  0 
irqbypass              13503  1 kvm

Comment 5 Sahid Ferdjaoui 2017-11-09 13:30:50 UTC

Can you ensure module kvm-{adm|intel} is enabled?

Is the compute node installed on a virtual machine "nested virtualization" ? in such case you should enable it with parameter "nested=1"

rmmod  kvm-{amd|intel} 
echo "options kvm-{amd|intel} nested=1" > /etc/modprobe.d/dist.conf
modprobe  kvm-{amd|intel}

Note You need to log in before you can comment on or make changes to this bug.