Created attachment 1412680 [details] log files from controller Description of problem: While trying to create encrypted volume from glance image the volume creation failed. Version-Release number of selected component (if applicable): puppet-cinder-12.3.1-0.20180222074326.18152ac.el7ost.noarch python2-cinderclient-3.5.0-0.20180211213738.1de605c.el7ost.noarch python-cinder-12.0.1-0.20180304104943.0983a42.el7ost.noarch openstack-cinder-12.0.1-0.20180304104943.0983a42.el7ost.noarch How reproducible: Configure encryption support for cinder and nova Steps to Reproduce: 1. volume create --type LUKS --image cirros --size 1 image_encrypt 2. 3. Actual results: +--------------------------------------+----------------+-----------+------+-------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+----------------+-----------+------+-------------+ | 4ff1d5b8-3255-4be8-ae34-1f8bfa6298c8 | image1_encrypt | error | 1 | | Expected results: status available Additional info: logs file attached to bug
This seems to indicate that the multipath command is not installed: 2018-03-25 08:26:16.122 29 ERROR cinder.volume.flows.manager.create_volume [req-dc2de0b0-1159-46cf-8749-264c720c862e efae9e8a2fe343d4be38936300efda6b 49bb8a31ac044c1c9337677a5cfdd748 - - -] Failed to copy image d0c1a936-62f5-4c7a-903d-4d1784145afe to volume: 4ff1d5b8-3255-4be8-ae34-1f8bfa6298c8: ExceptionChainer: Chained Exception #1 Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 462, in _detach_volume ignore_errors=ignore_errors) File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 137, in trace_logging_wrapper return f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner return f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 848, in disconnect_volume ignore_errors=ignore_errors) File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 899, in _cleanup_connection self._linuxscsi.flush_multipath_device(multipath_name) File "/usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py", line 275, in flush_multipath_device root_helper=self._root_helper) File "/usr/lib/python2.7/site-packages/os_brick/executor.py", line 52, in _execute result = self.__execute(*args, **kwargs) File "/usr/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py", line 187, in execute cmd=sanitized_cmd, description=six.text_type(e)) ProcessExecutionError: [Errno 2] No such file or directory Command: multipath -f 3514f0c5a51601682 Exit code: - Stdout: None Stderr: None
The os-brick package needs to depend on device-mapper-multipath.
Eric, FYI not sure this is multipath only. Failed to create an encrypted volume form an image on a simple LVM deployment. The traceback isn't exactly the same in my case. 'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9fe9b58f-5477-40f2-824e-113e43fc9c4a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None' 2018-04-01 07:37:23.637 46 WARNING cinder.volume.manager [req-50dc3d9d-f95f-4d8b-9403-a8233ac69447 ce31ffad3b064b3e86d8f85300e37b96 6dbc86f1432144e993360b8b4765ce9e - - -] Flow 'volume_create_manager' (8f80113a-19f0-4b08-bfc4-8b76cd45cb87) transitioned into state 'REVERTED' from state 'RUNNING' 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server [req-50dc3d9d-f95f-4d8b-9403-a8233ac69447 ce31ffad3b064b3e86d8f85300e37b96 6dbc86f1432144e993360b8b4765ce9e - - -] Exception during message handling: TypeError: 'NoneType' object is not iterable 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server File "<string>", line 2, in create_volume 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/objects/cleanable.py", line 207, in wrapper 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 668, in create_volume 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server _run_flow() 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 660, in _run_flow 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server flow_engine.run() 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server for _state in self.run_iter(timeout=timeout): 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server failure.Failure.reraise_if_any(er_failures) 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 336, in reraise_if_any 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server failures[0].reraise() 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 343, in reraise 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server six.reraise(*self._exc_info) 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server result = task.execute(**arguments) 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 972, in execute 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server **volume_spec) 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 881, in _create_from_image 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server image_service) 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server File "<string>", line 2, in _create_from_image_cache_or_download 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/coordination.py", line 151, in _synchronized 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server return f(*a, **k) 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 779, in _create_from_image_cache_or_download 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server backend_name) as tmp_image: 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__ 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server return self.gen.next() 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/image/image_utils.py", line 761, in fetch 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server fetch_verify_image(context, image_service, image_id, tmp) 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/image/image_utils.py", line 370, in fetch_verify_image 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server None, None) 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/image/image_utils.py", line 302, in fetch 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server image_service.download(context, image_id, image_file) 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/image/glance.py", line 329, in download 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server for chunk in image_chunks: 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not iterable 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server 2018-04-01 07:37:23.650 46 DEBUG cinder.volume.flows.manager.create_volume [req-50dc3d9d-f95f-4d8b-9403-a8233ac69447 ce31ffad3b064b3e86d8f85300e37b96 6dbc86f1432144e993360b8b4765ce9e - - -] Volume reschedule parameters: True retry: {u'num_attempts': 2, u'backends': [u'hostgroup@tripleo_iscsi#tripleo_iscsi', u'hostgroup@tripleo_iscsi#tripleo_iscsi'], u'hosts': [u'hostgroup@tripleo_iscsi#tripleo_iscsi', u'hostgroup@tripleo_iscsi#tripleo_iscsi']} get_flow /usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py:1102 2018-04-01 07:37:23.656 46 WARNING stevedore.named [req-50dc3d9d-f95f-4d8b-9403-a8233ac69447 ce31ffad3b064b3e86d8f85300e37b96 6dbc86f1432144e993360b8b4765ce9e - - -] Could not load cinder.keymgr.conf_key_mgr.ConfKeyManager 2018-04-01 07:37:23.656 46 WARNING castellan.key_manager [req-50dc3d9d-f95f-4d8b-9403-a8233ac69447 ce31ffad3b064b3e86d8f85300e37b96 6dbc86f1432144e993360b8b4765ce9e - - -] Deprecation Warning : cinder.keymgr.conf_key_mgr.ConfKeyManager is not a stevedore based driver, trying to load it as a class: NoMatches: No 'castellan.drivers' driver found, looking for 'cinder.keymgr.conf_key_mgr.ConfKeyManager' 2018-04-01 07:37:23.666 46 DEBUG cinder.volume.manager [req-50
Created attachment 1415755 [details] Cinder and Glance logs
BTW what does this mean why do we see it? cinder.keymgr.conf_key_mgr.ConfKeyManager is not a stevedore based driver, trying to load it as a class: NoMatches: No 'castellan.drivers' driver found, looking for 'cinder.keymgr.conf_key_mgr.ConfKeyManager'
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server File "/usr/lib/python2.7/site-packages/cinder/image/glance.py", line 329, in download 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server for chunk in image_chunks: 2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not iterable This seems to be because glanceclient can't find the image data -- it's a separate issue from what I was fixing here originally. cinder.keymgr.conf_key_mgr.ConfKeyManager is not a stevedore based driver, trying to load it as a class: NoMatches: No 'castellan.drivers' driver found, looking for 'cinder.keymgr.conf_key_mgr.ConfKeyManager' This is a warning message that can be ignored.
Verified on: python2-os-brick-2.3.1-1.el7ost.noarch I think fixed-in doesn't mention python2, suspect a typo, as my file name version are very close but I have no python-os-brick on my controller, only python2-os-brick.. Create new encrypted vol from image: #cinder create 1 --image cirros --volume-type LUKS --name EncVolFromImage Volume created and available: cinder list +--------------------------------------+-----------+-----------------+------+-------------+----------+--------------------------------------+ | ID | Status | Name | Size | Volume Type | Bootable | Attached to | +--------------------------------------+-----------+-----------------+------+-------------+----------+--------------------------------------+ .. .. | | 8489b592-b4dd-4ff9-8fc9-6d1e74dd26a6 | available | EncVolFromImage | 1 | LUKS | true | | +--------------------------------------+-----------+-----------------+------+-------------+----------+- Ran this a good few times already works every time. Works as expected.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2018:2086