Bug 1134982
| Summary: | Unable to delete instance after trying boot from encrypted volume when keymgr.fixed_key not defined | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | bkopilov <bkopilov> | ||||
| Component: | openstack-nova | Assignee: | Russell Bryant <rbryant> | ||||
| Status: | CLOSED INSUFFICIENT_DATA | QA Contact: | Ami Jeain <ajeain> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | low | ||||||
| Version: | 5.0 (RHEL 6) | CC: | bkopilov, dasmith, eharney, ndipanov, sgordon, yeylon | ||||
| Target Milestone: | --- | Keywords: | ZStream | ||||
| Target Release: | 6.0 (Juno) | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2014-10-30 15:18:38 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Traceback from nova.conf
2014-08-28 17:35:51.041 4228 AUDIT nova.virt.block_device [req-3efb617c-ceba-4bc2-956e-02e248d47049 3667d8514167424e80938f104e7c6cbc 0f34bc7346c44b788e8d31a8f4c17162] [instance: 2270c4f8-ae06-4a99-be8b-1c0cd1
90f9fd] Booting with volume 96365f29-e596-4ad7-98d5-500cb4f5f2b1 at vda
2014-08-28 17:35:51.042 4228 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.35.3.109
2014-08-28 17:35:51.194 4228 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.35.3.109
2014-08-28 17:35:51.622 4228 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.35.3.109
2014-08-28 17:35:51.965 4228 INFO nova.virt.libvirt.driver [req-3efb617c-ceba-4bc2-956e-02e248d47049 3667d8514167424e80938f104e7c6cbc 0f34bc7346c44b788e8d31a8f4c17162] [instance: 2270c4f8-ae06-4a99-be8b-1c0cd
190f9fd] Creating image
2014-08-28 17:35:51.966 4228 WARNING nova.virt.libvirt.driver [req-3efb617c-ceba-4bc2-956e-02e248d47049 3667d8514167424e80938f104e7c6cbc 0f34bc7346c44b788e8d31a8f4c17162] [instance: 2270c4f8-ae06-4a99-be8b-1c
0cd190f9fd] File injection into a boot from volume instance is not supported
2014-08-28 17:35:53.618 4228 AUDIT nova.compute.resource_tracker [-] Auditing locally available compute resources
2014-08-28 17:35:53.676 4228 AUDIT nova.compute.resource_tracker [-] Free ram (MB): 11679
2014-08-28 17:35:53.676 4228 AUDIT nova.compute.resource_tracker [-] Free disk (GB): -4
2014-08-28 17:35:53.676 4228 AUDIT nova.compute.resource_tracker [-] Free VCPUS: 4
2014-08-28 17:35:53.697 4228 INFO nova.compute.resource_tracker [-] Compute_service record updated for dhcp-3-109.tlv.redhat.com:dhcp-3-109.tlv.redhat.com
2014-08-28 17:35:53.984 4228 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.35.3.109
2014-08-28 17:35:54.027 4228 ERROR nova.compute.manager [req-3efb617c-ceba-4bc2-956e-02e248d47049 3667d8514167424e80938f104e7c6cbc 0f34bc7346c44b788e8d31a8f4c17162] [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190
f9fd] Instance failed to spawn
2014-08-28 17:35:54.027 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] Traceback (most recent call last):
2014-08-28 17:35:54.027 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1738, in _spawn
2014-08-28 17:35:54.027 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] block_device_info)
2014-08-28 17:35:54.027 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2289, in spawn
2014-08-28 17:35:54.027 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] block_device_info)
2014-08-28 17:35:54.027 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3664, in _create_domain_and
_network
2014-08-28 17:35:54.027 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] encryption)
2014-08-28 17:35:54.027 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1199, in _get_volume_encryp
tor
2014-08-28 17:35:54.027 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] **encryption)
2014-08-28 17:35:54.027 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/__init__.py", line 32, in get_volume_en
cryptor
2014-08-28 17:35:54.027 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] encryptor = nop.NoOpEncryptor(connection_info, **kwargs)
2014-08-28 17:35:54.027 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/nop.py", line 33, in __init__
2014-08-28 17:35:54.027 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] super(NoOpEncryptor, self).__init__(connection_info, **kwargs)
2014-08-28 17:35:54.027 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/base.py", line 38, in __init__
2014-08-28 17:35:54.027 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] self._key_manager = keymgr.API()
2014-08-28 17:35:54.027 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/keymgr/__init__.py", line 34, in API
2014-08-28 17:35:54.027 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] return cls()
2014-08-28 17:35:54.027 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/keymgr/conf_key_mgr.py", line 58, in __init__
2014-08-28 17:35:54.027 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] raise ValueError(_('keymgr.fixed_key not defined'))
2014-08-28 17:35:54.027 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] ValueError: keymgr.fixed_key not defined
2014-08-28 17:35:54.027 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd]
2014-08-28 17:35:54.076 4228 AUDIT nova.compute.manager [req-3efb617c-ceba-4bc2-956e-02e248d47049 3667d8514167424e80938f104e7c6cbc 0f34bc7346c44b788e8d31a8f4c17162] [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190
f9fd] Terminating instance
2014-08-28 17:35:54.084 4228 ERROR nova.virt.libvirt.driver [-] [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] During wait destroy, instance disappeared.
2014-08-28 17:35:54.086 4228 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.35.3.109
2014-08-28 17:35:54.140 4228 INFO nova.network.neutronv2.api [req-3efb617c-ceba-4bc2-956e-02e248d47049 3667d8514167424e80938f104e7c6cbc 0f34bc7346c44b788e8d31a8f4c17162] [instance: 2270c4f8-ae06-4a99-be8b-1c0
cd190f9fd] Unable to reset device ID for port None
2014-08-28 17:35:54.205 4228 ERROR nova.compute.manager [req-3efb617c-ceba-4bc2-956e-02e248d47049 3667d8514167424e80938f104e7c6cbc 0f34bc7346c44b788e8d31a8f4c17162] [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190
f9fd] Error: keymgr.fixed_key not defined
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] Traceback (most recent call last):
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1329, in _build_instance
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] set_access_ip=set_access_ip)
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 393, in decorated_function
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] return function(self, context, *args, **kwargs)
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1741, in _spawn
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] LOG.exception(_('Instance failed to spawn'), instance=instance)
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] six.reraise(self.type_, self.value, self.tb)
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1738, in _spawn
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] block_device_info)
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2289, in spawn
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] block_device_info)
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3664, in _create_domain_and
_network
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] encryption)
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1199, in _get_volume_encryp
tor
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] **encryption)
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/__init__.py", line 32, in get_volume_en
cryptor
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] encryptor = nop.NoOpEncryptor(connection_info, **kwargs)
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/nop.py", line 33, in __init__
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] super(NoOpEncryptor, self).__init__(connection_info, **kwargs)
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/base.py", line 38, in __init__
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] self._key_manager = keymgr.API()
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/keymgr/__init__.py", line 34, in API
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] return cls()
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/keymgr/conf_key_mgr.py", line 58, in __init__
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] raise ValueError(_('keymgr.fixed_key not defined'))
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] ValueError: keymgr.fixed_key not defined
2014-08-28 17:35:54.205 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd]
2014-08-28 17:35:54.327 4228 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: keymgr.fixed_key not defined
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/server.py", line 139, in inner
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher return func(*args, **kwargs)
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher payload)
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw)
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 274, in decorated_function
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher pass
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 260, in decorated_function
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 327, in decorated_function
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher function(self, context, *args, **kwargs)
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 303, in decorated_function
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher e, sys.exc_info())
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 290, in decorated_function
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2102, in run_instance
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher do_run_instance()
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line 249, in inner
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs)
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2101, in do_run_instance
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher legacy_bdm_in_spec)
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1225, in _run_instance
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher notify("error", fault=e) # notify that build failed
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1209, in _run_instance
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher instance, image_meta, legacy_bdm_in_spec)
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1373, in _build_instance
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher filter_properties, bdms, legacy_bdm_in_spec)
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1419, in _reschedule_or_error
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher self._log_original_error(exc_info, instance_uuid)
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1414, in _reschedule_or_error
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher bdms, requested_networks)
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2147, in _shutdown_instance
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher requested_networks)
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2137, in _shutdown_instance
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher block_device_info)
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 962, in destroy
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher destroy_disks, migrate_data)
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1052, in cleanup
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher encryption)
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1199, in _get_volume_encryptor
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher **encryption)
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/__init__.py", line 32, in get_volume_encryptor
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher encryptor = nop.NoOpEncryptor(connection_info, **kwargs)
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/nop.py", line 33, in __init__
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher super(NoOpEncryptor, self).__init__(connection_info, **kwargs)
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/base.py", line 38, in __init__
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher self._key_manager = keymgr.API()
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/keymgr/__init__.py", line 34, in API
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher return cls()
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/keymgr/conf_key_mgr.py", line 58, in __init__
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher raise ValueError(_('keymgr.fixed_key not defined'))
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher ValueError: keymgr.fixed_key not defined
2014-08-28 17:35:54.327 4228 TRACE oslo.messaging.rpc.dispatcher
2014-08-28 17:35:54.328 4228 ERROR oslo.messaging._drivers.common [-] Returning exception keymgr.fixed_key not defined to caller
2014-08-28 17:35:54.329 4228 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispat
ch_and_reply\n incoming.message))\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', '
File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/usr/lib/python2.7/site-packages/oslo
/messaging/rpc/server.py", line 139, in inner\n return func(*args, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped\n payload)\n', ' File "/usr/lib/python
2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapp
ed\n return f(self, context, *args, **kw)\n', ' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 274, in decorated_function\n pass\n', ' File "/usr/lib/python2.7/site-packages/nov
a/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 260, in decorated_function\
n return function(self, context, *args, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 327, in decorated_function\n function(self, context, *args, **kwargs)\n', '
File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 303, in decorated_function\n e, sys.exc_info())\n', ' File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", lin
e 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 290, in decorated_function\n return function(self, context,
*args, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2102, in run_instance\n do_run_instance()\n', ' File "/usr/lib/python2.7/site-packages/nova/openstack/common/l
ockutils.py", line 249, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2101, in do_run_instance\n legacy_bdm_in_spec)\n', ' File "/usr
/lib/python2.7/site-packages/nova/compute/manager.py", line 1225, in _run_instance\n notify("error", fault=e) # notify that build failed\n', ' File "/usr/lib/python2.7/site-packages/nova/openstack/common
/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1209, in _run_instance\n instance, image_m
eta, legacy_bdm_in_spec)\n', ' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1373, in _build_instance\n filter_properties, bdms, legacy_bdm_in_spec)\n', ' File "/usr/lib/python2.7
/site-packages/nova/compute/manager.py", line 1419, in _reschedule_or_error\n self._log_original_error(exc_info, instance_uuid)\n', ' File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.
py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1414, in _reschedule_or_error\n bdms, requested_netw
orks)\n', ' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2147, in _shutdown_instance\n requested_networks)\n', ' File "/usr/lib/python2.7/site-packages/nova/openstack/common/excu
tils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2137, in _shutdown_instance\n block_device_info
)\n', ' File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 962, in destroy\n destroy_disks, migrate_data)\n', ' File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py",
line 1052, in cleanup\n encryption)\n', ' File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1199, in _get_volume_encryptor\n **encryption)\n', ' File "/usr/lib/python2.7/site-
packages/nova/volume/encryptors/__init__.py", line 32, in get_volume_encryptor\n encryptor = nop.NoOpEncryptor(connection_info, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/nova/volume/encryptor
s/nop.py", line 33, in __init__\n super(NoOpEncryptor, self).__init__(connection_info, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/base.py", line 38, in __init__\n sel
f._key_manager = keymgr.API()\n', ' File "/usr/lib/python2.7/site-packages/nova/keymgr/__init__.py", line 34, in API\n return cls()\n', ' File "/usr/lib/python2.7/site-packages/nova/keymgr/conf_key_mgr.p
y", line 58, in __init__\n raise ValueError(_(\'keymgr.fixed_key not defined\'))\n', 'ValueError: keymgr.fixed_key not defined\n']
2014-08-28 17:36:04.658 4228 AUDIT nova.compute.manager [req-6c8fbba3-d701-4b5c-b5e9-f79b183042fd 3667d8514167424e80938f104e7c6cbc 0f34bc7346c44b788e8d31a8f4c17162] [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190
f9fd] Terminating instance
2014-08-28 17:36:04.664 4228 ERROR nova.virt.libvirt.driver [-] [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] During wait destroy, instance disappeared.
2014-08-28 17:36:04.667 4228 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.35.3.109
2014-08-28 17:36:04.747 4228 ERROR nova.compute.manager [req-6c8fbba3-d701-4b5c-b5e9-f79b183042fd 3667d8514167424e80938f104e7c6cbc 0f34bc7346c44b788e8d31a8f4c17162] [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190
f9fd] Setting instance vm_state to ERROR
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] Traceback (most recent call last):
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2264, in do_terminate_instance
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] self._delete_instance(context, instance, bdms, quotas)
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 103, in inner
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] rv = f(*args, **kwargs)
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2233, in _delete_instance
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] quotas.rollback()
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] six.reraise(self.type_, self.value, self.tb)
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2205, in _delete_instance
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] self._shutdown_instance(context, db_inst, bdms)
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2147, in _shutdown_instance
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] requested_networks)
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] six.reraise(self.type_, self.value, self.tb)
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2137, in _shutdown_instance
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] block_device_info)
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 962, in destroy
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] destroy_disks, migrate_data)
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1052, in cleanup
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] encryption)
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1199, in _get_volume_encryp
tor
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] **encryption)
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/__init__.py", line 32, in get_volume_en
cryptor
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] encryptor = nop.NoOpEncryptor(connection_info, **kwargs)
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/nop.py", line 33, in __init__
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] super(NoOpEncryptor, self).__init__(connection_info, **kwargs)
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/base.py", line 38, in __init__
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] self._key_manager = keymgr.API()
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/keymgr/__init__.py", line 34, in API
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] return cls()
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/keymgr/conf_key_mgr.py", line 58, in __init__
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] raise ValueError(_('keymgr.fixed_key not defined'))
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] ValueError: keymgr.fixed_key not defined
2014-08-28 17:36:04.747 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd]
2014-08-28 17:36:04.961 4228 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: keymgr.fixed_key not defined
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher payload)
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw)
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 274, in decorated_function
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher pass
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 260, in decorated_function
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 327, in decorated_function
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher function(self, context, *args, **kwargs)
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 303, in decorated_function
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher e, sys.exc_info())
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 290, in decorated_function
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2276, in terminate_instance
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher do_terminate_instance(instance, bdms)
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line 249, in inner
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs)
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2274, in do_terminate_instance
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher self._set_instance_error_state(context, instance['uuid'])
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2264, in do_terminate_instance
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher self._delete_instance(context, instance, bdms, quotas)
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 103, in inner
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher rv = f(*args, **kwargs)
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2233, in _delete_instance
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher quotas.rollback()
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2205, in _delete_instance
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher self._shutdown_instance(context, db_inst, bdms)
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2147, in _shutdown_instance
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher requested_networks)
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2137, in _shutdown_instance
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher block_device_info)
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 962, in destroy
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher destroy_disks, migrate_data)
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1052, in cleanup
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher encryption)
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1199, in _get_volume_encryptor
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher **encryption)
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/__init__.py", line 32, in get_volume_encryptor
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher encryptor = nop.NoOpEncryptor(connection_info, **kwargs)
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/nop.py", line 33, in __init__
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher super(NoOpEncryptor, self).__init__(connection_info, **kwargs)
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/base.py", line 38, in __init__
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher self._key_manager = keymgr.API()
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/keymgr/__init__.py", line 34, in API
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher return cls()
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/keymgr/conf_key_mgr.py", line 58, in __init__
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher raise ValueError(_('keymgr.fixed_key not defined'))
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher ValueError: keymgr.fixed_key not defined
2014-08-28 17:36:04.961 4228 TRACE oslo.messaging.rpc.dispatcher
2014-08-28 17:36:04.962 4228 ERROR oslo.messaging._drivers.common [-] Returning exception keymgr.fixed_key not defined to caller
2014-08-28 17:36:04.962 4228 ERROR oslo.messaging._drivers.common [-] ['Traceback (most recent call last):\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispat
ch_and_reply\n incoming.message))\n', ' File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch\n return self._do_dispatch(endpoint, method, ctxt, args)\n', '
File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch\n result = getattr(endpoint, method)(ctxt, **new_args)\n', ' File "/usr/lib/python2.7/site-packages/nova
/exception.py", line 88, in wrapped\n payload)\n', ' File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', '
File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped\n return f(self, context, *args, **kw)\n', ' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 274, in
decorated_function\n pass\n', ' File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/pyth
on2.7/site-packages/nova/compute/manager.py", line 260, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line
327, in decorated_function\n function(self, context, *args, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 303, in decorated_function\n e, sys.exc_info())\n', '
File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.7/site-packages/nova/compute/man
ager.py", line 290, in decorated_function\n return function(self, context, *args, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2276, in terminate_instance\n do_
terminate_instance(instance, bdms)\n', ' File "/usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line 249, in inner\n return f(*args, **kwargs)\n', ' File "/usr/lib/python2.7/site-pac
kages/nova/compute/manager.py", line 2274, in do_terminate_instance\n self._set_instance_error_state(context, instance[\'uuid\'])\n', ' File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutil
s.py", line 68, in __exit__\n six.reraise(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2264, in do_terminate_instance\n self._delete_inst
ance(context, instance, bdms, quotas)\n', ' File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 103, in inner\n rv = f(*args, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/nova/compute/m
anager.py", line 2233, in _delete_instance\n quotas.rollback()\n', ' File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise(self.type_, self.value
, self.tb)\n', ' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2205, in _delete_instance\n self._shutdown_instance(context, db_inst, bdms)\n', ' File "/usr/lib/python2.7/site-pack
ages/nova/compute/manager.py", line 2147, in _shutdown_instance\n requested_networks)\n', ' File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__\n six.reraise
(self.type_, self.value, self.tb)\n', ' File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2137, in _shutdown_instance\n block_device_info)\n', ' File "/usr/lib/python2.7/site-packages
/nova/virt/libvirt/driver.py", line 962, in destroy\n destroy_disks, migrate_data)\n', ' File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1052, in cleanup\n encryption)\n', '
File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1199, in _get_volume_encryptor\n **encryption)\n', ' File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/__init__.py", l
ine 32, in get_volume_encryptor\n encryptor = nop.NoOpEncryptor(connection_info, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/nop.py", line 33, in __init__\n super(NoOp
Encryptor, self).__init__(connection_info, **kwargs)\n', ' File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/base.py", line 38, in __init__\n self._key_manager = keymgr.API()\n', ' File "/usr
/lib/python2.7/site-packages/nova/keymgr/__init__.py", line 34, in API\n return cls()\n', ' File "/usr/lib/python2.7/site-packages/nova/keymgr/conf_key_mgr.py", line 58, in __init__\n raise ValueError(
_(\'keymgr.fixed_key not defined\'))\n', 'ValueError: keymgr.fixed_key not defined\n']
2014-08-28 17:36:12.423 4228 AUDIT nova.compute.manager [req-81897c89-18ee-4c73-941f-e266f0a07a16 3667d8514167424e80938f104e7c6cbc 0f34bc7346c44b788e8d31a8f4c17162] [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190
f9fd] Terminating instance
2014-08-28 17:36:12.428 4228 ERROR nova.virt.libvirt.driver [-] [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] During wait destroy, instance disappeared.
2014-08-28 17:36:12.431 4228 INFO urllib3.connectionpool [-] Starting new HTTP connection (1): 10.35.3.109
2014-08-28 17:36:12.505 4228 ERROR nova.compute.manager [req-81897c89-18ee-4c73-941f-e266f0a07a16 3667d8514167424e80938f104e7c6cbc 0f34bc7346c44b788e8d31a8f4c17162] [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190
f9fd] Setting instance vm_state to ERROR
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] Traceback (most recent call last):
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2264, in do_terminate_instance
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] self._delete_instance(context, instance, bdms, quotas)
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 103, in inner
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] rv = f(*args, **kwargs)
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2233, in _delete_instance
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] quotas.rollback()
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] six.reraise(self.type_, self.value, self.tb)
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2205, in _delete_instance
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] self._shutdown_instance(context, db_inst, bdms)
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2147, in _shutdown_instance
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] requested_networks)
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] six.reraise(self.type_, self.value, self.tb)
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2137, in _shutdown_instance
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] block_device_info)
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 962, in destroy
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] destroy_disks, migrate_data)
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1052, in cleanup
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] encryption)
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1199, in _get_volume_encryp
tor
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] **encryption)
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/__init__.py", line 32, in get_volume_en
cryptor
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] encryptor = nop.NoOpEncryptor(connection_info, **kwargs)
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/nop.py", line 33, in __init__
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] super(NoOpEncryptor, self).__init__(connection_info, **kwargs)
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/base.py", line 38, in __init__
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] self._key_manager = keymgr.API()
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/keymgr/__init__.py", line 34, in API
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] return cls()
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] File "/usr/lib/python2.7/site-packages/nova/keymgr/conf_key_mgr.py", line 58, in __init__
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] raise ValueError(_('keymgr.fixed_key not defined'))
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd] ValueError: keymgr.fixed_key not defined
2014-08-28 17:36:12.505 4228 TRACE nova.compute.manager [instance: 2270c4f8-ae06-4a99-be8b-1c0cd190f9fd]
2014-08-28 17:36:12.679 4228 ERROR oslo.messaging.rpc.dispatcher [-] Exception during message handling: keymgr.fixed_key not defined
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher Traceback (most recent call last):
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 133, in _dispatch_and_reply
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher incoming.message))
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 176, in _dispatch
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/oslo/messaging/rpc/dispatcher.py", line 122, in _do_dispatch
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher result = getattr(endpoint, method)(ctxt, **new_args)
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 88, in wrapped
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher payload)
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/exception.py", line 71, in wrapped
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher return f(self, context, *args, **kw)
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 274, in decorated_function
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher pass
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 260, in decorated_function
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 327, in decorated_function
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher function(self, context, *args, **kwargs)
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 303, in decorated_function
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher e, sys.exc_info())
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 290, in decorated_function
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2276, in terminate_instance
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher do_terminate_instance(instance, bdms)
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/lockutils.py", line 249, in inner
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher return f(*args, **kwargs)
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2274, in do_terminate_instance
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher self._set_instance_error_state(context, instance['uuid'])
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2264, in do_terminate_instance
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher self._delete_instance(context, instance, bdms, quotas)
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/hooks.py", line 103, in inner
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher rv = f(*args, **kwargs)
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2233, in _delete_instance
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher quotas.rollback()
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2205, in _delete_instance
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher self._shutdown_instance(context, db_inst, bdms)
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2147, in _shutdown_instance
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher requested_networks)
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2137, in _shutdown_instance
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher block_device_info)
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 962, in destroy
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher destroy_disks, migrate_data)
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1052, in cleanup
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher encryption)
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 1199, in _get_volume_encryptor
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher **encryption)
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/__init__.py", line 32, in get_volume_encryptor
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher encryptor = nop.NoOpEncryptor(connection_info, **kwargs)
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/nop.py", line 33, in __init__
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher super(NoOpEncryptor, self).__init__(connection_info, **kwargs)
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/volume/encryptors/base.py", line 38, in __init__
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher self._key_manager = keymgr.API()
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/keymgr/__init__.py", line 34, in API
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher return cls()
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/nova/keymgr/conf_key_mgr.py", line 58, in __init__
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher raise ValueError(_('keymgr.fixed_key not defined'))
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher ValueError: keymgr.fixed_key not defined
2014-08-28 17:36:12.679 4228 TRACE oslo.messaging.rpc.dispatcher
2014-08-28 17:36:12.681 4228 ERROR oslo.messaging._drivers.common [-] Returning exception keymgr.fixed_key not defined to caller
Setting keymgr.fixed_key is required for using encrypted volumes in general, but this does look like a problem. Assigning this to the Nova component since failure to delete the instance needs to be handled there first. Please attach the compute log as a file. Created attachment 932210 [details]
openstack logs
It looks to me like it is failing the delete because keymgr.fixed_key is not defined, is that right? Given that Eric confirms that that must be defined, does defining it allow the delete to proceed? Or does having it undefined get you into a situation where the instance is thereafter undeletable at all? Hi Dan , In case keymgr is not defined we get into undeletable instance . Benny Once your instance is undeletable, can you define the keymgr and then delete it? (In reply to Dan Smith from comment #7) > Once your instance is undeletable, can you define the keymgr and then delete > it? If you're able to update the bug with this requested feedback, feel free to reopen. Thanks! Hi , How can i configure keymgr ? Benny |
Description of problem: when creating encrypted volume with image and trying to boot nova from volume, instance state changed to Error and we are not able to delete this instance . Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: [root@dhcp-3-109 nova(keystone_admin)]# cinder encryption-type-create ENC_BENNY LuksEncryptor --key_size 512 --cipher aes-xts-plain64 +--------------------------------------+---------------+-----------------+----------+------------------+ | Volume Type ID | Provider | Cipher | Key Size | Control Location | +--------------------------------------+---------------+-----------------+----------+------------------+ | 000893d0-558d-4a1a-acf3-f6bc70b443a5 | LuksEncryptor | aes-xts-plain64 | 512 | front-end | +--------------------------------------+---------------+-----------------+----------+------------------+ [root@dhcp-3-109 nova(keystone_admin)]# glance image-list +--------------------------------------+---------+-------------+------------------+----------+--------+ | ID | Name | Disk Format | Container Format | Size | Status | +--------------------------------------+---------+-------------+------------------+----------+--------+ | ec50d68b-8df8-4c57-bdc7-d0505074855e | cirros1 | qcow2 | bare | 13147648 | active | | a33ec991-535b-4ef4-a9ce-f2f5fe9010dd | cirros2 | qcow2 | bare | 13147648 | active | +--------------------------------------+---------+-------------+------------------+----------+--------+ [root@dhcp-3-109 nova(keystone_admin)]# cinder create 1 --volume-type 000893d0-558d-4a1a-acf3-f6bc70b443a5 --image-id ec50d68b-8df8-4c57-bdc7-d0505074855e +---------------------+--------------------------------------+ | Property | Value | +---------------------+--------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | false | | created_at | 2014-08-28T14:32:59.339337 | | display_description | None | | display_name | None | | encrypted | True | | id | 96365f29-e596-4ad7-98d5-500cb4f5f2b1 | | image_id | ec50d68b-8df8-4c57-bdc7-d0505074855e | | metadata | {} | | size | 1 | | snapshot_id | None | | source_volid | None | | status | creating | | volume_type | ENC_BENNY | +---------------------+--------------------------------------+ [root@dhcp-3-109 nova(keystone_admin)]# [root@dhcp-3-109 nova(keystone_admin)]# cinder show 96365f29-e596-4ad7-98d5-500cb4f5f2b1 +--------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Property | Value | +--------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | attachments | [] | | availability_zone | nova | | bootable | true | | created_at | 2014-08-28T14:32:59.000000 | | display_description | None | | display_name | None | | encrypted | True | | id | 96365f29-e596-4ad7-98d5-500cb4f5f2b1 | | metadata | {} | | os-vol-host-attr:host | dhcp-3-109.tlv.redhat.com | | os-vol-mig-status-attr:migstat | None | | os-vol-mig-status-attr:name_id | None | | os-vol-tenant-attr:tenant_id | 0f34bc7346c44b788e8d31a8f4c17162 | | size | 1 | | snapshot_id | None | | source_volid | None | | status | available | | volume_image_metadata | {u'container_format': u'bare', u'min_ram': u'0', u'disk_format': u'qcow2', u'image_name': u'cirros1', u'image_id': u'ec50d68b-8df8-4c57-bdc7-d0505074855e', u'checksum': u'd972013792949d0d3ba628fbe8685bce', u'min_disk': u'0', u'size': u'13147648'} | | volume_type | ENC_BENNY | +--------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ ### boot nova from this volume from horizon , the instance is in error state and could not be deleted even after --state-reset in nova . Actual results: Expected results: Additional info: