Bug 1560244 - Encrypted volume from image failed to be created
Summary: Encrypted volume from image failed to be created
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-os-brick
Version: 13.0 (Queens)
Hardware: Unspecified
OS: Unspecified
urgent
urgent
Target Milestone: beta
: 13.0 (Queens)
Assignee: Eric Harney
QA Contact: Avi Avraham
Kim Nylander
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-03-25 09:00 UTC by Avi Avraham
Modified: 2018-06-27 13:49 UTC (History)
6 users (show)

Fixed In Version: python-os-brick-2.3.0-2.el7ost
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-06-27 13:48:49 UTC
Target Upstream Version:


Attachments (Terms of Use)
log files from controller (112.34 KB, application/x-gzip)
2018-03-25 09:00 UTC, Avi Avraham
no flags Details
Cinder and Glance logs (422.63 KB, application/zip)
2018-04-01 07:41 UTC, Tzach Shefi
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2018:2086 None None None 2018-06-27 13:49:42 UTC

Description Avi Avraham 2018-03-25 09:00:48 UTC
Created attachment 1412680 [details]
log files from controller

Description of problem:
While trying to create encrypted volume from glance image 
the volume creation failed.  

Version-Release number of selected component (if applicable):
puppet-cinder-12.3.1-0.20180222074326.18152ac.el7ost.noarch
python2-cinderclient-3.5.0-0.20180211213738.1de605c.el7ost.noarch
python-cinder-12.0.1-0.20180304104943.0983a42.el7ost.noarch
openstack-cinder-12.0.1-0.20180304104943.0983a42.el7ost.noarch

How reproducible:
Configure encryption support for cinder and nova 

Steps to Reproduce:
1. volume create --type LUKS --image cirros --size 1 image_encrypt
2.
3.

Actual results:
+--------------------------------------+----------------+-----------+------+-------------+
| ID                                   | Name           | Status    | Size | Attached to |
+--------------------------------------+----------------+-----------+------+-------------+
| 4ff1d5b8-3255-4be8-ae34-1f8bfa6298c8 | image1_encrypt | error     |    1 |             |


Expected results:
status available 

Additional info:
logs file attached to bug

Comment 1 Eric Harney 2018-03-26 13:23:44 UTC
This seems to indicate that the multipath command is not installed:

2018-03-25 08:26:16.122 29 ERROR cinder.volume.flows.manager.create_volume [req-dc2de0b0-1159-46cf-8749-264c720c862e efae9e8a2fe343d4be38936300efda6b 49bb8a31ac044c1c9337677a5cfdd748 - - -] Failed to copy image d0c1a936-62f5-4c7a-903d-4d1784145afe to volume: 4ff1d5b8-3255-4be8-ae34-1f8bfa6298c8: ExceptionChainer: 
Chained Exception #1
	Traceback (most recent call last):
	  File "/usr/lib/python2.7/site-packages/cinder/volume/driver.py", line 462, in _detach_volume
	    ignore_errors=ignore_errors)
	  File "/usr/lib/python2.7/site-packages/os_brick/utils.py", line 137, in trace_logging_wrapper
	    return f(*args, **kwargs)
	  File "/usr/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
	    return f(*args, **kwargs)
	  File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 848, in disconnect_volume
	    ignore_errors=ignore_errors)
	  File "/usr/lib/python2.7/site-packages/os_brick/initiator/connectors/iscsi.py", line 899, in _cleanup_connection
	    self._linuxscsi.flush_multipath_device(multipath_name)
	  File "/usr/lib/python2.7/site-packages/os_brick/initiator/linuxscsi.py", line 275, in flush_multipath_device
	    root_helper=self._root_helper)
	  File "/usr/lib/python2.7/site-packages/os_brick/executor.py", line 52, in _execute
	    result = self.__execute(*args, **kwargs)
	  File "/usr/lib/python2.7/site-packages/os_brick/privileged/rootwrap.py", line 187, in execute
	    cmd=sanitized_cmd, description=six.text_type(e))
	ProcessExecutionError: [Errno 2] No such file or directory
	Command: multipath -f 3514f0c5a51601682
	Exit code: -
	Stdout: None
	Stderr: None

Comment 2 Eric Harney 2018-03-29 12:29:11 UTC
The os-brick package needs to depend on device-mapper-multipath.

Comment 4 Tzach Shefi 2018-04-01 07:40:49 UTC
Eric, 

FYI not sure this is multipath only. 
Failed to create an encrypted volume form an image on a simple LVM deployment. The traceback isn't exactly the same in my case. 

'cinder.volume.flows.manager.create_volume.ExtractVolumeRefTask;volume:create' (9fe9b58f-5477-40f2-824e-113e43fc9c4a) transitioned into state 'REVERTED' from state 'REVERTING' with result 'None'
2018-04-01 07:37:23.637 46 WARNING cinder.volume.manager [req-50dc3d9d-f95f-4d8b-9403-a8233ac69447 ce31ffad3b064b3e86d8f85300e37b96 6dbc86f1432144e993360b8b4765ce9e - - -] Flow 'volume_create_manager' (8f80113a-19f0-4b08-bfc4-8b76cd45cb87) transitioned into state 'REVERTED' from state 'RUNNING'
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server [req-50dc3d9d-f95f-4d8b-9403-a8233ac69447 ce31ffad3b064b3e86d8f85300e37b96 6dbc86f1432144e993360b8b4765ce9e - - -] Exception during message handling: TypeError: 'NoneType' object is not iterable
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server   File "<string>", line 2, in create_volume
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/objects/cleanable.py", line 207, in wrapper
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server     result = f(*args, **kwargs)
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 668, in create_volume
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server     _run_flow()
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 660, in _run_flow
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server     flow_engine.run()
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server     for _state in self.run_iter(timeout=timeout):
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server     failure.Failure.reraise_if_any(er_failures)
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 336, in reraise_if_any
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server     failures[0].reraise()
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/taskflow/types/failure.py", line 343, in reraise
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server     six.reraise(*self._exc_info)
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server     result = task.execute(**arguments)
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 972, in execute
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server     **volume_spec)
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 881, in _create_from_image
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server     image_service)
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server   File "<string>", line 2, in _create_from_image_cache_or_download
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/coordination.py", line 151, in _synchronized
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server     return f(*a, **k)
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py", line 779, in _create_from_image_cache_or_download
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server     backend_name) as tmp_image:
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server   File "/usr/lib64/python2.7/contextlib.py", line 17, in __enter__
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server     return self.gen.next()
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/image/image_utils.py", line 761, in fetch
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server     fetch_verify_image(context, image_service, image_id, tmp)
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/image/image_utils.py", line 370, in fetch_verify_image
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server     None, None)
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/image/image_utils.py", line 302, in fetch
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server     image_service.download(context, image_id, image_file)
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/image/glance.py", line 329, in download
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server     for chunk in image_chunks:
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not iterable
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server 
2018-04-01 07:37:23.650 46 DEBUG cinder.volume.flows.manager.create_volume [req-50dc3d9d-f95f-4d8b-9403-a8233ac69447 ce31ffad3b064b3e86d8f85300e37b96 6dbc86f1432144e993360b8b4765ce9e - - -] Volume reschedule parameters: True retry: {u'num_attempts': 2, u'backends': [u'hostgroup@tripleo_iscsi#tripleo_iscsi', u'hostgroup@tripleo_iscsi#tripleo_iscsi'], u'hosts': [u'hostgroup@tripleo_iscsi#tripleo_iscsi', u'hostgroup@tripleo_iscsi#tripleo_iscsi']} get_flow /usr/lib/python2.7/site-packages/cinder/volume/flows/manager/create_volume.py:1102
2018-04-01 07:37:23.656 46 WARNING stevedore.named [req-50dc3d9d-f95f-4d8b-9403-a8233ac69447 ce31ffad3b064b3e86d8f85300e37b96 6dbc86f1432144e993360b8b4765ce9e - - -] Could not load cinder.keymgr.conf_key_mgr.ConfKeyManager
2018-04-01 07:37:23.656 46 WARNING castellan.key_manager [req-50dc3d9d-f95f-4d8b-9403-a8233ac69447 ce31ffad3b064b3e86d8f85300e37b96 6dbc86f1432144e993360b8b4765ce9e - - -] Deprecation Warning : cinder.keymgr.conf_key_mgr.ConfKeyManager is not a stevedore based driver, trying to load it as a class: NoMatches: No 'castellan.drivers' driver found, looking for 'cinder.keymgr.conf_key_mgr.ConfKeyManager'
2018-04-01 07:37:23.666 46 DEBUG cinder.volume.manager [req-50

Comment 5 Tzach Shefi 2018-04-01 07:41:24 UTC
Created attachment 1415755 [details]
Cinder and Glance logs

Comment 6 Tzach Shefi 2018-04-01 07:42:23 UTC
BTW what does this mean why do we see it? 

cinder.keymgr.conf_key_mgr.ConfKeyManager is not a stevedore based driver, trying to load it as a class: NoMatches: No 'castellan.drivers' driver found, looking for 'cinder.keymgr.conf_key_mgr.ConfKeyManager'

Comment 8 Eric Harney 2018-04-03 13:05:15 UTC
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server   File "/usr/lib/python2.7/site-packages/cinder/image/glance.py", line 329, in download
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server     for chunk in image_chunks:
2018-04-01 07:37:23.638 46 ERROR oslo_messaging.rpc.server TypeError: 'NoneType' object is not iterable

This seems to be because glanceclient can't find the image data -- it's a separate issue from what I was fixing here originally.


cinder.keymgr.conf_key_mgr.ConfKeyManager is not a stevedore based driver, trying to load it as a class: NoMatches: No 'castellan.drivers' driver found, looking for 'cinder.keymgr.conf_key_mgr.ConfKeyManager'

This is a warning message that can be ignored.

Comment 13 Tzach Shefi 2018-05-01 19:51:36 UTC
Verified on: 
python2-os-brick-2.3.1-1.el7ost.noarch

I think fixed-in doesn't mention python2, suspect a typo, as my file name version are very close but I have no python-os-brick on my controller, only python2-os-brick.. 

Create new encrypted vol from image:

#cinder create 1 --image cirros --volume-type LUKS --name EncVolFromImage

Volume created and available:

cinder list
+--------------------------------------+-----------+-----------------+------+-------------+----------+--------------------------------------+
| ID                                   | Status    | Name            | Size | Volume Type | Bootable | Attached to                          |
+--------------------------------------+-----------+-----------------+------+-------------+----------+--------------------------------------+
..
..                                    |
| 8489b592-b4dd-4ff9-8fc9-6d1e74dd26a6 | available | EncVolFromImage | 1    | LUKS        | true     |                                      |
+--------------------------------------+-----------+-----------------+------+-------------+----------+-


Ran this a good few times already works every time. 
Works as expected.

Comment 15 errata-xmlrpc 2018-06-27 13:48:49 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:2086


Note You need to log in before you can comment on or make changes to this bug.