Bug 1798890
| Summary: | [RHOS16.1] encrypted volumes can only be uploaded as raw, but the api lets you try to convert on upload and fail | |||
|---|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | bkopilov <bkopilov> | |
| Component: | openstack-cinder | Assignee: | Brian Rosmaita <brian.rosmaita> | |
| Status: | CLOSED ERRATA | QA Contact: | Evelina Shames <eshames> | |
| Severity: | low | Docs Contact: | Andy Stillman <astillma> | |
| Priority: | low | |||
| Version: | 16.0 (Train) | CC: | eharney, gfidente, ltoscano | |
| Target Milestone: | z9 | Keywords: | Triaged, ZStream | |
| Target Release: | 16.1 (Train on RHEL 8.2) | |||
| Hardware: | x86_64 | |||
| OS: | Linux | |||
| Whiteboard: | ||||
| Fixed In Version: | openstack-cinder-15.4.0-1.20220810154916.58f0e73.el8ost | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 2078008 (view as bug list) | Environment: | ||
| Last Closed: | 2022-12-07 20:24:45 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 2078008 | |||
| Bug Blocks: | ||||
|
Description
bkopilov
2020-02-06 08:58:01 UTC
More information: the upload-to-image used by default RAW image and as far as i know not supported on ceph, I tried to convert it to qcow2 /home/heat-admin ==== controller-2 controllers ==== /var/log/containers/cinder/cinder-volume.log:397:2020-02-06 09:15:21.610 13026 ERROR cinder.volume.manager [req-205a21f8-7471-48f9-8c45-eb1e1a830de1 99beb5bee51240bcbd39e74ad53eda96 2b46203ca51e4ff3afc1a6734d663c8a - default default] Upload volume to image encountered an error (image-id: a6c29719-a05b-4cbf-97e5-b0f7b3ff6b36).: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. /var/log/containers/cinder/cinder-volume.log:399:2020-02-06 09:15:21.702 13026 WARNING cinder.volume.manager [req-205a21f8-7471-48f9-8c45-eb1e1a830de1 99beb5bee51240bcbd39e74ad53eda96 2b46203ca51e4ff3afc1a6734d663c8a - default default] Deleting image in unexpected status: queued.: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. /var/log/containers/cinder/cinder-volume.log:400:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server [req-205a21f8-7471-48f9-8c45-eb1e1a830de1 99beb5bee51240bcbd39e74ad53eda96 2b46203ca51e4ff3afc1a6734d663c8a - default default] Exception during message handling: oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. /var/log/containers/cinder/cinder-volume.log:405:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server Traceback (most recent call last): /var/log/containers/cinder/cinder-volume.log:406:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming /var/log/containers/cinder/cinder-volume.log:407:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) /var/log/containers/cinder/cinder-volume.log:408:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch /var/log/containers/cinder/cinder-volume.log:409:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) /var/log/containers/cinder/cinder-volume.log:410:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch /var/log/containers/cinder/cinder-volume.log:411:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) /var/log/containers/cinder/cinder-volume.log:412:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/volume/manager.py", line 1687, in copy_volume_to_image /var/log/containers/cinder/cinder-volume.log:413:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server payload['message'] = six.text_type(error) /var/log/containers/cinder/cinder-volume.log:414:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ /var/log/containers/cinder/cinder-volume.log:415:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server self.force_reraise() /var/log/containers/cinder/cinder-volume.log:416:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise /var/log/containers/cinder/cinder-volume.log:417:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) /var/log/containers/cinder/cinder-volume.log:418:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise /var/log/containers/cinder/cinder-volume.log:419:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server raise value /var/log/containers/cinder/cinder-volume.log:420:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/volume/manager.py", line 1668, in copy_volume_to_image /var/log/containers/cinder/cinder-volume.log:421:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server image_service, image_meta) /var/log/containers/cinder/cinder-volume.log:422:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/volume/drivers/rbd.py", line 1595, in copy_volume_to_image /var/log/containers/cinder/cinder-volume.log:423:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server image_meta, tmp_file) /var/log/containers/cinder/cinder-volume.log:424:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/image/image_utils.py", line 700, in upload_volume /var/log/containers/cinder/cinder-volume.log:425:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server compress=compress) /var/log/containers/cinder/cinder-volume.log:426:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/image/image_utils.py", line 328, in convert_image /var/log/containers/cinder/cinder-volume.log:427:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server compress=compress) /var/log/containers/cinder/cinder-volume.log:428:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/image/image_utils.py", line 271, in _convert_image /var/log/containers/cinder/cinder-volume.log:429:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server utils.execute(*cmd, run_as_root=run_as_root) /var/log/containers/cinder/cinder-volume.log:430:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/cinder/utils.py", line 126, in execute /var/log/containers/cinder/cinder-volume.log:431:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server return processutils.execute(*cmd, **kwargs) /var/log/containers/cinder/cinder-volume.log:432:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server File "/usr/lib/python3.6/site-packages/oslo_concurrency/processutils.py", line 424, in execute /var/log/containers/cinder/cinder-volume.log:433:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server cmd=sanitized_cmd) /var/log/containers/cinder/cinder-volume.log:434:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server oslo_concurrency.processutils.ProcessExecutionError: Unexpected error while running command. /var/log/containers/cinder/cinder-volume.log:435:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server Command: qemu-img convert -O qcow2 -c /var/lib/cinder/conversion/volume-eb097bfa-b81f-424c-b20a-c040d131aec1-a6c29719-a05b-4cbf-97e5-b0f7b3ff6b36 /var/lib/cinder/conversion/tmp8p1lxuvi /var/log/containers/cinder/cinder-volume.log:436:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server Exit code: 1 /var/log/containers/cinder/cinder-volume.log:437:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server Stdout: '' /var/log/containers/cinder/cinder-volume.log:438:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server Stderr: "qemu-img: Could not open '/var/lib/cinder/conversion/volume-eb097bfa-b81f-424c-b20a-c040d131aec1-a6c29719-a05b-4cbf-97e5-b0f7b3ff6b36': Parameter 'key-secret' is required for cipher\n" /var/log/containers/cinder/cinder-volume.log:439:2020-02-06 09:15:22.770 13026 ERROR oslo_messaging.rpc.server (In reply to Eric Harney from comment #2) > Comment #1 is a whole separate issue from the title/description of this bz. > Can you split it into another BZ? Hi Eric , Here is an email: Brian Rosmaita Thu, Feb 6, 3:56 PM (4 days ago) to me, Luigi, Eric, Brian, rhos-cinder, Tzach Apologies for top-posting. It think we have 3 different bugs here: 1. booting an instance in Nova from a Glance image created from an encrypted volume by the Cinder upload-volume-to-image action is *not* supported by Nova. The bug is that instead of letting the instance go to ACTIVE, Nova should return some kind of error (maybe at the Compute API layer; if the image has the cinder_encryption_key_id metadata on it, you can't boot from it). 2. trying to upload an encrypted volume as an image as qcow2 is failing (need to check with Eric about whether this is supported or not). 3. Possible data loss from createImage action -- hopefully this is not something people are likely to do, but if someone does the unsupported Nova action described in #1, Benny verified that even though the instance isn't usable, you can do the 'nova image-create' action on it. This results in another image (presumable unusable) that has the same cinder_encryption_key_id and cinder_encryption_key_deletion_policy as the original image -- so when this useless image is deleted, the key for the usable image is deleted. Bug #3 is really bad. It can be fixed short-term by a configuration change in Nova, namely, by including the cinder_encryption_key_* in the Nova non_inheritable_image_properties list. (Longer term, it would be fixed by the fix to #1, which wouldn't let this case happen.) I've posted some additional BZs to split this up. Using the numbers from comment #3: 1. https://bugzilla.redhat.com/show_bug.cgi?id=1801282 -- (nova API change to prevent boot of image created from encrypted cinder volume) 2. this bug (encrypted volumes can only be uploaded as raw, but the api lets you try to convert on upload and fail) 3. https://bugzilla.redhat.com/show_bug.cgi?id=1801255 -- blacklist cinder_encryption_key_* properties so that nova doesn't put them on images) Comment #6 addressed the needinfo for this bug. Fix is in openstack-cinder-15.4.0-1.20220810154916.58f0e73.el8ost, which has the rhos-16.1-rhel-8-trunk-candidate tag: https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=2118316 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1.9 bug fix and enhancement advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:8795 |