Bug 1081015
Summary: | failed to launch an instance from ISO image | ||
---|---|---|---|
Product: | [Community] RDO | Reporter: | Yogev Rabl <yrabl> |
Component: | openstack-nova | Assignee: | RHOS Maint <rhos-maint> |
Status: | CLOSED INSUFFICIENT_DATA | QA Contact: | Ami Jeain <ajeain> |
Severity: | urgent | Docs Contact: | |
Priority: | urgent | ||
Version: | unspecified | CC: | bderzhavets, dallan, ndipanov, rbryant, sross, yeylon, yrabl |
Target Milestone: | --- | Keywords: | TestBlocker |
Target Release: | --- | ||
Hardware: | x86_64 | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | Bug Fix | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2014-06-18 07:31:46 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
Yogev Rabl
2014-03-26 13:38:29 UTC
In addition, I fail to launch an instance from a volume. The log shows a similar error: 2014-03-26 17:24:31.731 534 ERROR nova.compute.manager [req-340eedb3-8e8c-4a97-9920-58e06b70b741 bae9e2b600104c748a7fc7da0eafef65 011bc60688114a2fadf33fcb5a9b1503] [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] Instance failed to spawn 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] Traceback (most recent call last): 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1473, in _spawn 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] block_device_info) 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2230, in spawn 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] admin_pass=admin_password) 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2634, in _create_image 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] net = netutils.get_injected_network_template(network_info) 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/nova/virt/netutils.py", line 71, in get_injected_network_template 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] if not (network_info and template): 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 379, in __len__ 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] return self._sync_wrapper(fn, *args, **kwargs) 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 366, in _sync_wrapper 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] self.wait() 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 398, in wait 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] self[:] = self._gt.wait() 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 168, in wait 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] return self._exit_event.wait() 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 116, in wait 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] return hubs.get_hub().switch() 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 187, in switch 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] return self.greenlet.switch() 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] result = function(*args, **kwargs) 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1254, in _allocate_network_async 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] dhcp_options=dhcp_options) 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 94, in wrapped 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] return func(self, context, *args, **kwargs) 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 48, in wrapper 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] res = f(self, context, *args, **kwargs) 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 302, in allocate_for_instance 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] nw_info = self.network_rpcapi.allocate_for_instance(context, **args) 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/nova/network/rpcapi.py", line 170, in allocate_for_instance 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] macs=jsonutils.to_primitive(macs)) 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 150, in call 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] wait_for_reply=True, timeout=timeout) 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] timeout=timeout) 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 409, in send 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] return self._send(target, ctxt, message, wait_for_reply, timeout) 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 400, in _send 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] result = self._waiter.wait(msg_id, timeout) 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 280, in wait 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] reply, ending, trylock = self._poll_queue(msg_id, timeout) 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 220, in _poll_queue 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] message = self.waiters.get(msg_id, timeout) 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 126, in get 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] 'to message ID %s' % msg_id) 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] MessagingTimeout: Timed out waiting for a reply to message ID 2fad0329cfec4bd684fab6508dafd8aa 2014-03-26 17:24:31.731 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] 2014-03-26 17:24:31.759 534 DEBUG nova.compute.claims [req-340eedb3-8e8c-4a97-9920-58e06b70b741 bae9e2b600104c748a7fc7da0eafef65 011bc60688114a2fadf33fcb5a9b1503] [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] Aborting claim: [Claim: 2048 MB memory, 60 GB disk, 2 VCPUS] abort /usr/lib/python2.7/site-packages/nova/compute/claims.py:113 2014-03-26 17:24:31.761 534 DEBUG nova.openstack.common.lockutils [req-340eedb3-8e8c-4a97-9920-58e06b70b741 bae9e2b600104c748a7fc7da0eafef65 011bc60688114a2fadf33fcb5a9b1503] Got semaphore "compute_resources" lock /usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py:168 2014-03-26 17:24:31.762 534 DEBUG nova.openstack.common.lockutils [req-340eedb3-8e8c-4a97-9920-58e06b70b741 bae9e2b600104c748a7fc7da0eafef65 011bc60688114a2fadf33fcb5a9b1503] Got semaphore / lock "abort_instance_claim" inner /usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py:248 2014-03-26 17:24:31.763 534 DEBUG oslo.messaging._drivers.amqpdriver [-] MSG_ID is 78405a822ee74d8bacf616b5a4d67498 _send /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:377 2014-03-26 17:24:31.763 534 DEBUG oslo.messaging._drivers.amqp [-] UNIQUE_ID is e3088ba7befd4e4b8bf15ac7c0777fe7. _add_unique_id /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqp.py:338 2014-03-26 17:24:31.845 534 DEBUG nova.openstack.common.lockutils [req-340eedb3-8e8c-4a97-9920-58e06b70b741 bae9e2b600104c748a7fc7da0eafef65 011bc60688114a2fadf33fcb5a9b1503] Semaphore / lock released "abort_instance_claim" inner /usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py:252 2014-03-26 17:24:31.847 534 DEBUG oslo.messaging._drivers.amqpdriver [-] MSG_ID is cd23874ca52d473193bc6dfc4e3eb4c4 _send /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:377 2014-03-26 17:24:31.848 534 DEBUG oslo.messaging._drivers.amqp [-] UNIQUE_ID is bd9dfb96addc44449813eb0abf90e0bb. _add_unique_id /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqp.py:338 2014-03-26 17:24:31.899 534 DEBUG nova.compute.utils [req-340eedb3-8e8c-4a97-9920-58e06b70b741 bae9e2b600104c748a7fc7da0eafef65 011bc60688114a2fadf33fcb5a9b1503] [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] Timed out waiting for a reply to message ID 2fad0329cfec4bd684fab6508dafd8aa notify_about_instance_usage /usr/lib/python2.7/site-packages/nova/compute/utils.py:335 2014-03-26 17:24:31.899 534 TRACE nova.compute.utils [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] None 2014-03-26 17:24:31.899 534 TRACE nova.compute.utils [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] 2014-03-26 17:24:31.900 534 DEBUG nova.compute.manager [req-340eedb3-8e8c-4a97-9920-58e06b70b741 bae9e2b600104c748a7fc7da0eafef65 011bc60688114a2fadf33fcb5a9b1503] [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] Clean up resource before rescheduling. _reschedule_or_error /usr/lib/python2.7/site-packages/nova/compute/manager.py:1134 2014-03-26 17:24:31.901 534 AUDIT nova.compute.manager [req-340eedb3-8e8c-4a97-9920-58e06b70b741 bae9e2b600104c748a7fc7da0eafef65 011bc60688114a2fadf33fcb5a9b1503] [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] Terminating instance 2014-03-26 17:24:31.902 534 DEBUG oslo.messaging._drivers.amqpdriver [-] MSG_ID is 4c7d3470f80547c9b35cb72c8aa66c93 _send /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:377 2014-03-26 17:24:31.902 534 DEBUG oslo.messaging._drivers.amqp [-] UNIQUE_ID is 802d9ac739874f7b9968cdb56495f97b. _add_unique_id /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqp.py:338 2014-03-26 17:24:31.997 534 DEBUG nova.objects.instance [req-340eedb3-8e8c-4a97-9920-58e06b70b741 bae9e2b600104c748a7fc7da0eafef65 011bc60688114a2fadf33fcb5a9b1503] Lazy-loading `system_metadata' on Instance uuid e6c89aa2-50bd-4401-8a4c-1f229b5d935c obj_load_attr /usr/lib/python2.7/site-packages/nova/objects/instance.py:518 2014-03-26 17:24:31.998 534 DEBUG oslo.messaging._drivers.amqpdriver [-] MSG_ID is 3c8c4f089d51413c81f134655a7feb75 _send /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:377 2014-03-26 17:24:31.998 534 DEBUG oslo.messaging._drivers.amqp [-] UNIQUE_ID is ff1d81a6ea5b4f35a9a33abbb3a6288c. _add_unique_id /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqp.py:338 2014-03-26 17:24:32.103 534 DEBUG oslo.messaging._drivers.amqpdriver [-] MSG_ID is 98ae91b3f01249f79d6498af6f62674b _send /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:377 2014-03-26 17:24:32.104 534 DEBUG oslo.messaging._drivers.amqp [-] UNIQUE_ID is e0d3e87f130b4d738501691f758ebf0b. _add_unique_id /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqp.py:338 2014-03-26 17:24:32.630 534 DEBUG oslo.messaging._drivers.amqpdriver [-] MSG_ID is 430154bacc1f407f93ab0d9f175f81f6 _send /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:377 2014-03-26 17:24:32.631 534 DEBUG oslo.messaging._drivers.amqp [-] UNIQUE_ID is af3e879a6a054eb5961ce69b79fac865. _add_unique_id /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqp.py:338 2014-03-26 17:24:42.630 534 DEBUG oslo.messaging._drivers.amqpdriver [-] MSG_ID is 40f8c6ad7d2a4e9080f05636c50b4e5c _send /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:377 2014-03-26 17:24:42.631 534 DEBUG oslo.messaging._drivers.amqp [-] UNIQUE_ID is 4d14d8170d384d6c8e83922b06e17612. _add_unique_id /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqp.py:338 2014-03-26 17:24:52.632 534 DEBUG oslo.messaging._drivers.amqpdriver [-] MSG_ID is c08606c12f8649409a108cfca2c20102 _send /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:377 2014-03-26 17:24:52.633 534 DEBUG oslo.messaging._drivers.amqp [-] UNIQUE_ID is d4f7cf4a115e430e9efc30c7ac618a8e. _add_unique_id /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqp.py:338 2014-03-26 17:25:02.632 534 DEBUG oslo.messaging._drivers.amqpdriver [-] MSG_ID is a64376542e394678945bc46efc9fcea3 _send /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:377 2014-03-26 17:25:02.633 534 DEBUG oslo.messaging._drivers.amqp [-] UNIQUE_ID is 66fb356e2c9b4eb4bf428273109a6857. _add_unique_id /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqp.py:338 2014-03-26 17:25:12.633 534 DEBUG oslo.messaging._drivers.amqpdriver [-] MSG_ID is 98be923c246c4440ab8df181d0266032 _send /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:377 2014-03-26 17:25:12.634 534 DEBUG oslo.messaging._drivers.amqp [-] UNIQUE_ID is 502f1af4d0144a9288a672f2116dc8ac. _add_unique_id /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqp.py:338 2014-03-26 17:25:22.635 534 DEBUG oslo.messaging._drivers.amqpdriver [-] MSG_ID is 8b2cd24900814292a4c1e5b3864ea8a7 _send /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py:377 2014-03-26 17:25:22.636 534 DEBUG oslo.messaging._drivers.amqp [-] UNIQUE_ID is e5b8d87c368f4eb78c8433d6ee375003. _add_unique_id /usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqp.py:338 2014-03-26 17:25:32.111 534 ERROR nova.compute.manager [req-340eedb3-8e8c-4a97-9920-58e06b70b741 bae9e2b600104c748a7fc7da0eafef65 011bc60688114a2fadf33fcb5a9b1503] [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] Error: Timed out waiting for a reply to message ID 2fad0329cfec4bd684fab6508dafd8aa 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] Traceback (most recent call last): 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1066, in _build_instance 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] set_access_ip=set_access_ip) 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 360, in decorated_function 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] return function(self, context, *args, **kwargs) 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1476, in _spawn 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] LOG.exception(_('Instance failed to spawn'), instance=instance) 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__ 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] six.reraise(self.type_, self.value, self.tb) 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1473, in _spawn 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] block_device_info) 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2230, in spawn 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] admin_pass=admin_password) 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2634, in _create_image 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] net = netutils.get_injected_network_template(network_info) 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/nova/virt/netutils.py", line 71, in get_injected_network_template 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] if not (network_info and template): 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 379, in __len__ 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] return self._sync_wrapper(fn, *args, **kwargs) 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 366, in _sync_wrapper 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] self.wait() 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/nova/network/model.py", line 398, in wait 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] self[:] = self._gt.wait() 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 168, in wait 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] return self._exit_event.wait() 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 116, in wait 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] return hubs.get_hub().switch() 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/eventlet/hubs/hub.py", line 187, in switch 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] return self.greenlet.switch() 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 194, in main 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] result = function(*args, **kwargs) 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1254, in _allocate_network_async 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] dhcp_options=dhcp_options) 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 94, in wrapped 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] return func(self, context, *args, **kwargs) 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 48, in wrapper 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] res = f(self, context, *args, **kwargs) 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/nova/network/api.py", line 302, in allocate_for_instance 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] nw_info = self.network_rpcapi.allocate_for_instance(context, **args) 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/nova/network/rpcapi.py", line 170, in allocate_for_instance 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] macs=jsonutils.to_primitive(macs)) 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/client.py", line 150, in call 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] wait_for_reply=True, timeout=timeout) 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/oslo/messaging/transport.py", line 90, in _send 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] timeout=timeout) 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 409, in send 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] return self._send(target, ctxt, message, wait_for_reply, timeout) 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 400, in _send 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] result = self._waiter.wait(msg_id, timeout) 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 280, in wait 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] reply, ending, trylock = self._poll_queue(msg_id, timeout) 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 220, in _poll_queue 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] message = self.waiters.get(msg_id, timeout) 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] File "/usr/lib/python2.7/site-packages/oslo/messaging/_drivers/amqpdriver.py", line 126, in get 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] 'to message ID %s' % msg_id) 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] MessagingTimeout: Timed out waiting for a reply to message ID 2fad0329cfec4bd684fab6508dafd8aa 2014-03-26 17:25:32.111 534 TRACE nova.compute.manager [instance: e6c89aa2-50bd-4401-8a4c-1f229b5d935c] How can I check if this problem is been caused by the qpid? What is the command you used to launch the instance? I did all the steps in the horizon. (In reply to Yogev Rabl from comment #1) > In addition, I fail to launch an instance from a volume. The log shows a > similar error: Cannot reproduce. Launching instance via cinder volume ( on glusterfs ) [root@dfw02 ~(keystone_admin)]$ uname -a Linux dfw02.localdomain 3.13.9-200.fc20.x86_64 #1 SMP Fri Apr 4 12:13:05 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux [root@dfw02 ~(keystone_admin)]$ cinder create --image-id a6e8ef59-e492-46e2-8147-fd8b1a65ed73 --display_name CentOS65VLG0412 17 +---------------------+--------------------------------------+ | Property | Value | +---------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | created_at | 2014-04-13T09:23:37.319614 | | display_description | None | | display_name | CentOS65VLG0412 | | id | e2eb5e89-ca42-4342-a97c-2202370792dd | | image_id | a6e8ef59-e492-46e2-8147-fd8b1a65ed73 | | metadata | {} | | size | 17 | | snapshot_id | None | | source_volid | None | | status | creating | | volume_type | None | +---------------------+--------------------------------------+ [root@dfw02 ~(keystone_admin)]$ cinder list +--------------------------------------+-------------+-----------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-------------+-----------------+------+-------------+----------+-------------+ | 0f4bd623-9012-430a-8fd3-8fda7d710c81 | available | Win2012VLG01 | 20 | glusterfs | true | | | e2eb5e89-ca42-4342-a97c-2202370792dd | downloading | CentOS65VLG0412 | 17 | None | false | | +--------------------------------------+-------------+-----------------+------+-------------+----------+-------------+ [root@dfw02 ~(keystone_admin)]$ cinder list +--------------------------------------+-----------+-----------------+------+-------------+----------+-------------+ | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+-----------------+------+-------------+----------+-------------+ | 0f4bd623-9012-430a-8fd3-8fda7d710c81 | available | Win2012VLG01 | 20 | glusterfs | true | | | e2eb5e89-ca42-4342-a97c-2202370792dd | available | CentOS65VLG0412 | 17 | None | true | | +--------------------------------------+-----------+-----------------+------+-------------+----------+-------------+ [root@dfw02 ~(keystone_admin)]$ nova boot --flavor 2 --key_name oskey12 --block_device_mapping vda=e2eb5e89-ca42-4342-a97c-2202370792dd:::0 CentOS65RS01 +--------------------------------------+----------------------------------------------------+ | Property | Value | +--------------------------------------+----------------------------------------------------+ | OS-EXT-STS:task_state | scheduling | | image | Attempt to boot from volume - no image supplied | | OS-EXT-STS:vm_state | building | | OS-EXT-SRV-ATTR:instance_name | instance-000000f0 | | OS-SRV-USG:launched_at | None | | flavor | m1.small | | id | 03f08456-4500-4926-8657-3e7bed520faf | | security_groups | [{u'name': u'default'}] | | user_id | 970ed56ef7bc41d59c54f5ed8a1690dc | | OS-DCF:diskConfig | MANUAL | | accessIPv4 | | | accessIPv6 | | | progress | 0 | | OS-EXT-STS:power_state | 0 | | OS-EXT-AZ:availability_zone | nova | | config_drive | | | status | BUILD | | updated | 2014-04-13T09:41:10Z | | hostId | | | OS-EXT-SRV-ATTR:host | None | | OS-SRV-USG:terminated_at | None | | key_name | oskey12 | | OS-EXT-SRV-ATTR:hypervisor_hostname | None | | name | CentOS65RS01 | | adminPass | rtLyeoDuE4o8 | | tenant_id | d0a0acfdb62b4cc8a2bfa8d6a08bb62f | | created | 2014-04-13T09:41:10Z | | os-extended-volumes:volumes_attached | [{u'id': u'e2eb5e89-ca42-4342-a97c-2202370792dd'}] | | metadata | {} | +--------------------------------------+----------------------------------------------------+ [root@dfw02 ~(keystone_admin)]$ nova list +--------------------------------------+--------------+--------+------------+-------------+--------------+ | ID | Name | Status | Task State | Power State | Networks | +--------------------------------------+--------------+--------+------------+-------------+--------------+ | 03f08456-4500-4926-8657-3e7bed520faf | CentOS65RS01 | ACTIVE | None | Running | int=10.0.0.2 | +--------------------------------------+--------------+--------+------------+-------------+--------------+ [root@dfw02 ~(keystone_admin)]$ neutron floatingip-create ext Created a new floatingip: +---------------------+--------------------------------------+ | Field | Value | +---------------------+--------------------------------------+ | fixed_ip_address | | | floating_ip_address | 192.168.1.103 | | floating_network_id | 780ce2f3-2e6e-4881-bbac-857813f9a8e0 | | id | 7aa59bfe-43e9-41ec-8f02-d066a7fca92f | | port_id | | | router_id | | | tenant_id | d0a0acfdb62b4cc8a2bfa8d6a08bb62f | +---------------------+--------------------------------------+ [root@dfw02 ~(keystone_admin)]$ neutron port-list --device-id 03f08456-4500-4926-8657-3e7bed520faf +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+ | id | name | mac_address | fixed_ips | +--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+ | 25c35735-8054-45fe-972c-5b471221c389 | | fa:16:3e:99:c5:7b | {"subnet_id": "fa930cea-3d51-4cbe-a305-579f12aa53c0", "ip_address": "10.0.0.2"} | +--------------------------------------+------+-------------------+------------- [root@dfw02 ~(keystone_admin)]$ neutron floatingip-associate 7aa59bfe-43e9-41ec-8f02-d066a7fca92f 25c35735-8054-45fe-972c-5b471221c389 Associated floatingip 7aa59bfe-43e9-41ec-8f02-d066a7fca92f [root@dfw02 ~(keystone_admin)]$ ping 192.168.1.103 PING 192.168.1.103 (192.168.1.103) 56(84) bytes of data. 64 bytes from 192.168.1.103: icmp_seq=1 ttl=63 time=14.0 ms 64 bytes from 192.168.1.103: icmp_seq=2 ttl=63 time=1.78 ms ...... [root@dfw02 ~(keystone_admin)]$ ssh -l cloud-user -i oskey12.priv 192.168.1.103 The authenticity of host '192.168.1.103 (192.168.1.103)' can't be established. RSA key fingerprint is ca:73:c0:f7:3f:d3:b8:58:43:61:83:9e:bc:23:31:49. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '192.168.1.103' (RSA) to the list of known hosts. [cloud-user@centos65rs01 ~]$ uname -a Linux centos65rs01 2.6.32-431.3.1.el6.x86_64 #1 SMP Fri Jan 3 21:39:27 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux [cloud-user@centos65rs01 ~]$ ifconfig eth0 Link encap:Ethernet HWaddr FA:16:3E:99:C5:7B inet addr:10.0.0.2 Bcast:10.0.0.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fe99:c57b/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1454 Metric:1 RX packets:265 errors:0 dropped:0 overruns:0 frame:0 TX packets:248 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:29970 (29.2 KiB) TX bytes:26380 (25.7 KiB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) How did you install RDO on Fedora 20? I got errors from missing f20 repositories for a couple of the Foreman repos and the openstack-havana repo. When I hacked the repo files to use the f19 versions, I got a failure to install nova due to python-cmd2 being too old for python-cliff. I can hack around this problem too, but I want to make sure that my hacks are in sync with yours, and that the bug is not caused by such installation hacking. I've installed the RDO with packstack. Yogev, are you still able to repro this? Closing as insufficient data; if this behavior reappears, please reopen. The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days |