Bug 1827522

Summary: [RHOS16] Create volume failed to copy metadata to volume when upload-to-image
Product: Red Hat OpenStack Reporter: bkopilov <bkopilov>
Component: openstack-cinderAssignee: Brian Rosmaita <brian.rosmaita>
Status: ASSIGNED --- QA Contact: Evelina Shames <eshames>
Severity: high Docs Contact: Andy Stillman <astillma>
Priority: medium    
Version: 16.0 (Train)CC: aefrat, aruffin, averi, brian.rosmaita, cylopez, dhill, eharney, gfidente, jgrosso, jhardee, ltoscano, msecaur, nnavarat, omcgonag, pambre, rdiwakar, yocha
Target Milestone: z10Keywords: Triaged
Target Release: 16.1 (Train on RHEL 8.2)   
Hardware: x86_64   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description bkopilov 2020-04-24 05:17:31 UTC
Description of problem:

cinder create 1 --volume-type LUKS --image-id 81318302-674c-4714-95d1-65dfec8ab645

(overcloud) [stack@undercloud-0 ~]$ cinder show afc89b6b-989d-4603-a943-479636da30f7
+--------------------------------+-------------------------------------------------+
| Property                       | Value                                           |
+--------------------------------+-------------------------------------------------+
| attached_servers               | []                                              |
| attachment_ids                 | []                                              |
| availability_zone              | nova                                            |
| bootable                       | true                                            |
| consistencygroup_id            | None                                            |
| created_at                     | 2020-04-24T05:04:38.000000                      |
| description                    | None                                            |
| encrypted                      | True                                            |
| id                             | afc89b6b-989d-4603-a943-479636da30f7            |
| metadata                       |                                                 |
| migration_status               | None                                            |
| multiattach                    | False                                           |
| name                           | None                                            |
| os-vol-host-attr:host          | hostgroup@tripleo_iscsi#tripleo_iscsi           |
| os-vol-mig-status-attr:migstat | None                                            |
| os-vol-mig-status-attr:name_id | None                                            |
| os-vol-tenant-attr:tenant_id   | 8315d137369a4a919e56fab848822f5d                |
| replication_status             | None                                            |
| size                           | 1                                               |
| snapshot_id                    | None                                            |
| source_volid                   | None                                            |
| status                         | available                                       |
| updated_at                     | 2020-04-24T05:04:54.000000                      |
| user_id                        | 8e51a25213c2479abc3553e894638548                |
| volume_image_metadata          | checksum : ba3cd24377dde5dfdd58728894004abb     |
|                                | container_format : bare                         |
|                                | disk_format : raw                               |
|                                | image_id : 81318302-674c-4714-95d1-65dfec8ab645 |
|                                | image_name : cirros                             |
|                                | min_disk : 0                                    |
|                                | min_ram : 0                                     |
|                                | signature_verified : False                      |
|                                | size : 46137344                                 |
| volume_type                    | LUKS                                            |
+--------------------------------+-------------------------------------------------+

(overcloud) [stack@undercloud-0 ~]$ cinder  upload-to-image afc89b6b-989d-4603-a943-479636da30f7 UPLOAD_TO_IMAGE
+---------------------+--------------------------------------+
| Property            | Value                                |
+---------------------+--------------------------------------+
| container_format    | bare                                 |
| disk_format         | raw                                  |
| display_description | None                                 |
| id                  | afc89b6b-989d-4603-a943-479636da30f7 |
| image_id            | 19e66045-fef8-48d0-a1aa-be840308e8f4 |
| image_name          | UPLOAD_TO_IMAGE                      |
| protected           | False                                |
| size                | 1                                    |
| status              | uploading                            |
| updated_at          | 2020-04-24T05:04:54.000000           |
| visibility          | shared                               |
| volume_type         | LUKS                                 |
+---------------------+--------------------------------------+

+------------------------------------------------------------------------+------+---------------------------+--------+-----------------------------------------+-----------+------------+-------------+------+------------+
| Secret href                                                            | Name | Created                   | Status | Content types                           | Algorithm | Bit length | Secret type | Mode | Expiration |
+------------------------------------------------------------------------+------+---------------------------+--------+-----------------------------------------+-----------+------------+-------------+------+------------+
| http://10.0.0.112:9311/v1/secrets/621011be-668b-4d75-a1e5-8eaeaefe3b05 | None | 2020-04-24T05:04:36+00:00 | ACTIVE | {'default': 'application/octet-stream'} | aes       |        256 | symmetric   | None | None       |
| http://10.0.0.112:9311/v1/secrets/f89db57c-a44a-44ea-b7cc-60399ec562f9 | None | 2020-04-24T05:07:07+00:00 | ACTIVE | {'default': 'application/octet-stream'} | aes       |        256 | symmetric   | None | None       |
+------------------------------------------------------------------------+------+---------------------------+--------+-----------------------------------------+-----------+------------+-------------+------+------------+


(overcloud) [stack@undercloud-0 ~]$ glance image-show 19e66045-fef8-48d0-a1aa-be840308e8f4
+---------------------------------------+----------------------------------------------------------------------------------+
| Property                              | Value                                                                            |
+---------------------------------------+----------------------------------------------------------------------------------+
| checksum                              | 0762951ce262997974db2d0c9f82e475                                                 |
| cinder_encryption_key_deletion_policy | on_image_deletion                                                                |
| cinder_encryption_key_id              | f89db57c-a44a-44ea-b7cc-60399ec562f9                                             |
| container_format                      | bare                                                                             |
| created_at                            | 2020-04-24T05:07:08Z                                                             |
| direct_url                            | swift+config://ref1/glance/19e66045-fef8-48d0-a1aa-be840308e8f4                  |
| disk_format                           | raw                                                                              |
| id                                    | 19e66045-fef8-48d0-a1aa-be840308e8f4                                             |
| min_disk                              | 0                                                                                |
| min_ram                               | 0                                                                                |
| name                                  | UPLOAD_TO_IMAGE                                                                  |
| os_hash_algo                          | sha512                                                                           |
| os_hash_value                         | 835901a5407a357898ea9e70541e9e912c6eadad1a49aefe6e744c446482368c7180fab392ed40b4 |
|                                       | 817c32f8d2ec6c2e8c2284bd2c376e09ef72f3271e94992c                                 |
| os_hidden                             | False                                                                            |
| owner                                 | 8315d137369a4a919e56fab848822f5d                                                 |
| protected                             | False                                                                            |
| signature_verified                    | False                                                                            |
| size                                  | 1073741824                                                                       |
| status                                | active                                                                           |
| stores                                | default_backend                                                                  |
| tags                                  | []                                                                               |
| updated_at                            | 2020-04-24T05:07:19Z                                                             |
| virtual_size                          | Not available                                                                    |
| visibility                            | shared                                                                           |
+---------------------------------------+----------------------------------------------------------------------------------+



Try to create volume :
cinder create 2  --image-id 19e66045-fef8-48d0-a1aa-be840308e8f4

/var/log/containers/cinder/cinder-volume.log:57:2020-04-24 05:10:39.063 51 ERROR cinder.volume.flows.manager.create_volume [req-55bb6d9c-63cc-4338-beee-6f00ac45a476 8e51a25213c2479abc3553e894638548 8315d137369a4a919e56fab848822f5d - default default] Failed updating volume 85038fee-0765-4743-b74d-afc2a64c84f7 metadata using the provided image 19e66045-fef8-48d0-a1aa-be840308e8f4 metadata: cinder.exception.GlanceMetadataExists: Glance metadata cannot be updated, key signature_verified exists for volume id 85038fee-0765-4743-b74d-afc2a64c84f7
/var/log/containers/cinder/cinder-volume.log:58:2020-04-24 05:10:39.063 51 ERROR cinder.volume.flows.manager.create_volume Traceback (most recent call last):
/var/log/containers/cinder/cinder-volume.log:59:2020-04-24 05:10:39.063 51 ERROR cinder.volume.flows.manager.create_volume   File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 443, in _handle_bootable_volume_glance_meta
/var/log/containers/cinder/cinder-volume.log:60:2020-04-24 05:10:39.063 51 ERROR cinder.volume.flows.manager.create_volume     image_id, image_meta)
/var/log/containers/cinder/cinder-volume.log:61:2020-04-24 05:10:39.063 51 ERROR cinder.volume.flows.manager.create_volume   File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 658, in _capture_volume_image_metadata
/var/log/containers/cinder/cinder-volume.log:62:2020-04-24 05:10:39.063 51 ERROR cinder.volume.flows.manager.create_volume     volume_metadata)
/var/log/containers/cinder/cinder-volume.log:63:2020-04-24 05:10:39.063 51 ERROR cinder.volume.flows.manager.create_volume   File "/usr/lib/python3.6/site-packages/cinder/db/api.py", line 975, in volume_glance_metadata_bulk_create
/var/log/containers/cinder/cinder-volume.log:64:2020-04-24 05:10:39.063 51 ERROR cinder.volume.flows.manager.create_volume     metadata)
/var/log/containers/cinder/cinder-volume.log:65:2020-04-24 05:10:39.063 51 ERROR cinder.volume.flows.manager.create_volume   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 189, in wrapper
/var/log/containers/cinder/cinder-volume.log:66:2020-04-24 05:10:39.063 51 ERROR cinder.volume.flows.manager.create_volume     return f(*args, **kwargs)
/var/log/containers/cinder/cinder-volume.log:67:2020-04-24 05:10:39.063 51 ERROR cinder.volume.flows.manager.create_volume   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 204, in wrapper
/var/log/containers/cinder/cinder-volume.log:68:2020-04-24 05:10:39.063 51 ERROR cinder.volume.flows.manager.create_volume     return f(context, volume_id, *args, **kwargs)
/var/log/containers/cinder/cinder-volume.log:69:2020-04-24 05:10:39.063 51 ERROR cinder.volume.flows.manager.create_volume   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 5046, in volume_glance_metadata_bulk_create
/var/log/containers/cinder/cinder-volume.log:70:2020-04-24 05:10:39.063 51 ERROR cinder.volume.flows.manager.create_volume     volume_id=volume_id)
/var/log/containers/cinder/cinder-volume.log:71:2020-04-24 05:10:39.063 51 ERROR cinder.volume.flows.manager.create_volume cinder.exception.GlanceMetadataExists: Glance metadata cannot be updated, key signature_verified exists for volume id 85038fee-0765-4743-b74d-afc2a64c84f7
/var/log/containers/cinder/cinder-volume.log:72:2020-04-24 05:10:39.063 51 ERROR cinder.volume.flows.manager.create_volume 
/var/log/containers/cinder/cinder-volume.log:80:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager Traceback (most recent call last):
/var/log/containers/cinder/cinder-volume.log:81:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 443, in _handle_bootable_volume_glance_meta
/var/log/containers/cinder/cinder-volume.log:82:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager     image_id, image_meta)
/var/log/containers/cinder/cinder-volume.log:83:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 658, in _capture_volume_image_metadata
/var/log/containers/cinder/cinder-volume.log:84:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager     volume_metadata)
/var/log/containers/cinder/cinder-volume.log:85:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/cinder/db/api.py", line 975, in volume_glance_metadata_bulk_create
/var/log/containers/cinder/cinder-volume.log:86:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager     metadata)
/var/log/containers/cinder/cinder-volume.log:87:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 189, in wrapper
/var/log/containers/cinder/cinder-volume.log:88:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager     return f(*args, **kwargs)
/var/log/containers/cinder/cinder-volume.log:89:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 204, in wrapper
/var/log/containers/cinder/cinder-volume.log:90:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager     return f(context, volume_id, *args, **kwargs)
/var/log/containers/cinder/cinder-volume.log:91:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 5046, in volume_glance_metadata_bulk_create
/var/log/containers/cinder/cinder-volume.log:92:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager     volume_id=volume_id)
/var/log/containers/cinder/cinder-volume.log:93:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager cinder.exception.GlanceMetadataExists: Glance metadata cannot be updated, key signature_verified exists for volume id 85038fee-0765-4743-b74d-afc2a64c84f7
/var/log/containers/cinder/cinder-volume.log:94:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager 
/var/log/containers/cinder/cinder-volume.log:95:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager During handling of the above exception, another exception occurred:
/var/log/containers/cinder/cinder-volume.log:96:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager 
/var/log/containers/cinder/cinder-volume.log:97:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager Traceback (most recent call last):
/var/log/containers/cinder/cinder-volume.log:98:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
/var/log/containers/cinder/cinder-volume.log:99:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager     result = task.execute(**arguments)
/var/log/containers/cinder/cinder-volume.log:100:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 1135, in execute
/var/log/containers/cinder/cinder-volume.log:101:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager     **volume_spec)
/var/log/containers/cinder/cinder-volume.log:102:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/cinder/utils.py", line 696, in _wrapper
/var/log/containers/cinder/cinder-volume.log:103:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager     return r.call(f, *args, **kwargs)
/var/log/containers/cinder/cinder-volume.log:104:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call
/var/log/containers/cinder/cinder-volume.log:105:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager     return attempt.get(self._wrap_exception)
/var/log/containers/cinder/cinder-volume.log:106:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get
/var/log/containers/cinder/cinder-volume.log:107:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager     six.reraise(self.value[0], self.value[1], self.value[2])
/var/log/containers/cinder/cinder-volume.log:108:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
/var/log/containers/cinder/cinder-volume.log:109:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager     raise value
/var/log/containers/cinder/cinder-volume.log:110:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call
/var/log/containers/cinder/cinder-volume.log:111:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
/var/log/containers/cinder/cinder-volume.log:112:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 1039, in _create_from_image
/var/log/containers/cinder/cinder-volume.log:113:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager     image_meta=image_meta)
/var/log/containers/cinder/cinder-volume.log:114:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager   File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 453, in _handle_bootable_volume_glance_meta
/var/log/containers/cinder/cinder-volume.log:115:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager     raise exception.MetadataCopyFailure(reason=ex)
/var/log/containers/cinder/cinder-volume.log:116:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager cinder.exception.MetadataCopyFailure: Failed to copy metadata to volume: Glance metadata cannot be updated, key signature_verified exists for volume id 85038fee-0765-4743-b74d-afc2a64c84f7
/var/log/containers/cinder/cinder-volume.log:117:2020-04-24 05:10:39.070 51 ERROR cinder.volume.manager 
/var/log/containers/cinder/cinder-volume.log:125:2020-04-24 05:10:39.082 51 DEBUG cinder.volume.flows.common [req-55bb6d9c-63cc-4338-beee-6f00ac45a476 8e51a25213c2479abc3553e894638548 8315d137369a4a919e56fab848822f5d - default default] Setting Volume 85038fee-0765-4743-b74d-afc2a64c84f7 to error due to: Unknown reason error_out /usr/lib/python3.6/site-packages/cinder/volume/flows/common.py:83
/var/log/containers/cinder/cinder-volume.log:126:2020-04-24 05:10:39.088 51 ERROR cinder.volume.flows.manager.create_volume [req-55bb6d9c-63cc-4338-beee-6f00ac45a476 8e51a25213c2479abc3553e894638548 8315d137369a4a919e56fab848822f5d - default default] Volume 85038fee-0765-4743-b74d-afc2a64c84f7: create failed
/var/log/containers/cinder/cinder-volume.log:131:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server [req-55bb6d9c-63cc-4338-beee-6f00ac45a476 8e51a25213c2479abc3553e894638548 8315d137369a4a919e56fab848822f5d - default default] Exception during message handling: cinder.exception.MetadataCopyFailure: Failed to copy metadata to volume: Glance metadata cannot be updated, key signature_verified exists for volume id 85038fee-0765-4743-b74d-afc2a64c84f7
/var/log/containers/cinder/cinder-volume.log:132:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
/var/log/containers/cinder/cinder-volume.log:133:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 443, in _handle_bootable_volume_glance_meta
/var/log/containers/cinder/cinder-volume.log:134:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server     image_id, image_meta)
/var/log/containers/cinder/cinder-volume.log:135:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 658, in _capture_volume_image_metadata
/var/log/containers/cinder/cinder-volume.log:136:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server     volume_metadata)
/var/log/containers/cinder/cinder-volume.log:137:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/db/api.py", line 975, in volume_glance_metadata_bulk_create
/var/log/containers/cinder/cinder-volume.log:138:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server     metadata)
/var/log/containers/cinder/cinder-volume.log:139:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 189, in wrapper
/var/log/containers/cinder/cinder-volume.log:140:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
/var/log/containers/cinder/cinder-volume.log:141:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 204, in wrapper
/var/log/containers/cinder/cinder-volume.log:142:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server     return f(context, volume_id, *args, **kwargs)
/var/log/containers/cinder/cinder-volume.log:143:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/db/sqlalchemy/api.py", line 5046, in volume_glance_metadata_bulk_create
/var/log/containers/cinder/cinder-volume.log:144:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server     volume_id=volume_id)
/var/log/containers/cinder/cinder-volume.log:145:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server cinder.exception.GlanceMetadataExists: Glance metadata cannot be updated, key signature_verified exists for volume id 85038fee-0765-4743-b74d-afc2a64c84f7
/var/log/containers/cinder/cinder-volume.log:146:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server 
/var/log/containers/cinder/cinder-volume.log:147:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred:
/var/log/containers/cinder/cinder-volume.log:148:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server 
/var/log/containers/cinder/cinder-volume.log:149:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
/var/log/containers/cinder/cinder-volume.log:150:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming
/var/log/containers/cinder/cinder-volume.log:151:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
/var/log/containers/cinder/cinder-volume.log:152:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch
/var/log/containers/cinder/cinder-volume.log:153:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
/var/log/containers/cinder/cinder-volume.log:154:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch
/var/log/containers/cinder/cinder-volume.log:155:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
/var/log/containers/cinder/cinder-volume.log:156:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server   File "<decorator-gen-233>", line 2, in create_volume
/var/log/containers/cinder/cinder-volume.log:157:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/objects/cleanable.py", line 212, in wrapper
/var/log/containers/cinder/cinder-volume.log:158:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server     result = f(*args, **kwargs)
/var/log/containers/cinder/cinder-volume.log:159:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/volume/manager.py", line 749, in create_volume
/var/log/containers/cinder/cinder-volume.log:160:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server     _run_flow()
/var/log/containers/cinder/cinder-volume.log:161:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/volume/manager.py", line 741, in _run_flow
/var/log/containers/cinder/cinder-volume.log:162:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server     flow_engine.run()
/var/log/containers/cinder/cinder-volume.log:163:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/taskflow/engines/action_engine/engine.py", line 247, in run
/var/log/containers/cinder/cinder-volume.log:164:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server     for _state in self.run_iter(timeout=timeout):
/var/log/containers/cinder/cinder-volume.log:165:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/taskflow/engines/action_engine/engine.py", line 340, in run_iter
/var/log/containers/cinder/cinder-volume.log:166:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server     failure.Failure.reraise_if_any(er_failures)
/var/log/containers/cinder/cinder-volume.log:167:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/taskflow/types/failure.py", line 339, in reraise_if_any
/var/log/containers/cinder/cinder-volume.log:168:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server     failures[0].reraise()
/var/log/containers/cinder/cinder-volume.log:169:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/taskflow/types/failure.py", line 346, in reraise
/var/log/containers/cinder/cinder-volume.log:170:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server     six.reraise(*self._exc_info)
/var/log/containers/cinder/cinder-volume.log:171:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
/var/log/containers/cinder/cinder-volume.log:172:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server     raise value
/var/log/containers/cinder/cinder-volume.log:173:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/taskflow/engines/action_engine/executor.py", line 53, in _execute_task
/var/log/containers/cinder/cinder-volume.log:174:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server     result = task.execute(**arguments)
/var/log/containers/cinder/cinder-volume.log:175:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 1135, in execute
/var/log/containers/cinder/cinder-volume.log:176:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server     **volume_spec)
/var/log/containers/cinder/cinder-volume.log:177:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/utils.py", line 696, in _wrapper
/var/log/containers/cinder/cinder-volume.log:178:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server     return r.call(f, *args, **kwargs)
/var/log/containers/cinder/cinder-volume.log:179:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/retrying.py", line 223, in call
/var/log/containers/cinder/cinder-volume.log:180:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server     return attempt.get(self._wrap_exception)
/var/log/containers/cinder/cinder-volume.log:181:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/retrying.py", line 261, in get
/var/log/containers/cinder/cinder-volume.log:182:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server     six.reraise(self.value[0], self.value[1], self.value[2])
/var/log/containers/cinder/cinder-volume.log:183:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/six.py", line 693, in reraise
/var/log/containers/cinder/cinder-volume.log:184:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server     raise value
/var/log/containers/cinder/cinder-volume.log:185:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/retrying.py", line 217, in call
/var/log/containers/cinder/cinder-volume.log:186:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server     attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
/var/log/containers/cinder/cinder-volume.log:187:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 1039, in _create_from_image
/var/log/containers/cinder/cinder-volume.log:188:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server     image_meta=image_meta)
/var/log/containers/cinder/cinder-volume.log:189:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server   File "/usr/lib/python3.6/site-packages/cinder/volume/flows/manager/create_volume.py", line 453, in _handle_bootable_volume_glance_meta
/var/log/containers/cinder/cinder-volume.log:190:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server     raise exception.MetadataCopyFailure(reason=ex)
/var/log/containers/cinder/cinder-volume.log:191:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server cinder.exception.MetadataCopyFailure: Failed to copy metadata to volume: Glance metadata cannot be updated, key signature_verified exists for volume id 85038fee-0765-4743-b74d-afc2a64c84f7
/var/log/containers/cinder/cinder-volume.log:192:2020-04-24 05:10:39.099 51 ERROR oslo_messaging.rpc.server

Comment 10 Brian Rosmaita 2021-03-31 13:02:50 UTC
The short-term workaround for this is to remove the signature_verified metadata from any glance images.  This should not be on the image in the first place, it's a marker from cinder that the glance image contained the img_signature* properties [0] and that when the image was written to a volume, the signature was verified.  What happens is that if you turn on signature verification in cinder, it will fail volume creation when the img_signature* properties exist but the verification fails, and it will put "signature_verified: false" on volumes created from images that didn't have img_signature* metadata, and put "signature_verified: true" if verification passed.

What I'm getting at is that signature_verified is not an attribute of the image, it's an attribute of the volume, so removing it from images will not affect security or data integrity.  (Unless an operator happens to be using an image property with that name for some other purpose.)

So an automatic short-term workaround could be a cron job removing signature_verified from any glance image that has it.



[0] https://docs.openstack.org/glance/latest/user/signature.html

Comment 11 Matthew Secaur 2021-03-31 15:20:11 UTC
Hi, Brian,

Having that automated workaround is really helpful. Thank you for that. For our partners who provide this to their customers, though, I'm not sure it's a very reliable workaround longer-term.

What needs to happen to make a formal, permanent fix?

Thanks!

Comment 12 Brian Rosmaita 2021-03-31 20:23:22 UTC
(In reply to Matthew Secaur from comment #11)
> What needs to happen to make a formal, permanent fix?

I think it has 3 parts:

(1) introduce a non_inheritable_image_properties config option (like Nova has) to filter out properties that are in  the cinder volume_image_metadata of a volume, but which should not be put on the image when the volume is uploaded as an image to Glance.  One of these would be signature_verified, but also any of the img_signature* properties (because in most cases, the image1 -> volume -> image2 transformation will not give you an image2 that can be verified by the signature for image1).  Maybe this doesn't need to be a config option, but it would give operators some flexibility for using third-party tools.  Or maybe it could be a combination config opt + hard-coded list (which is what Nova has since ussuri).

(2) refactor how cinder handles the volume_image_metadata to include (1) but also there's a weird mix in there of accessing the DB directly and using cinder-oslo-versioned-objects to manipulate the image metadata (which is actually the reason for this bug).  Additionally it's not clear that we should be storing cinder-specific info (like signature_verified) in the volume_image_metadata as a conceptual matter (though that won't matter so much if (1) is implemented).

(3) introduce a filter similar to (1) except that it will filter out image properties that will not be stored in the volume_image_metadata in the first place.  This is needed to deal with "legacy" images that have signature_verified on them, but it might also be useful to make it a config option like (1).  Not sure about that, though.

Comment 13 youngcheol 2021-04-28 04:48:46 UTC
Hi team,

CU is wondering if it is possible not using "signature verified" that before get patch for this issue.

Could you help me for answering to CU?

Thank you,
YoungCheol.

Comment 14 Brian Rosmaita 2021-05-04 17:53:25 UTC
(In reply to youngcheol from comment #13)

> CU is wondering if it is possible not using "signature verified" that before
> get patch for this issue.

Yes.  In the cinder.conf file used by the cinder volume service, set

  verify_glance_signatures = disabled

What will happen is that if the image has the 'signature_verified' property, it will be included in the volume_image_metadata of the created volume, but we won't hit this bug because cinder will not try to update the value.

Comment 20 Brian Rosmaita 2022-09-16 03:40:39 UTC
I was asked to answer a question about the security implications of one of the workarounds for this bug.

The workaround is to set verify_glance_signatures=disabled in cinder.conf (which basically turns off the feature).

The security implication is that the data downloaded to cinder (and that cinder writes to the volume) is not verified by using the cryptographic signature.  But whether this is an issue depends on your use case, because the data *is* verified by other means.

Each image downloaded from glance to cinder (since Rocky) is verified using cryptographically strong checksumming (before that, image downloads were verified using md5; that's still the case for "legacy" images in a cloud that were created before Rocky).  This is the glance 'multihash' feature, which uses the os_hash_algo and os_hash_value image properties to store the checksum algorithm and hash.  The default is sha512.

So even with verify_glance_signatures=disabled, we have assurance that the image data originally uploaded to glance (when the os_hash_value was computed), is unchanged when it is consumed by cinder.  (The checksumming code, by the way, is not in cinder; it's in the python-glanceclient (since version 2.13.0), which cinder uses to download the image data.)

The use case addressed by image signature verification is to allow users to verify that a previously uploaded image has not been modified.  This can be done without the signature if a user computes the sha512 hash of the image before upload, and then verifies that the os_hash_value that glance sets on the image is the same.  Then you have to trust that cinder and nova are verifying the hash correctly when consuming the image, but you'd also have to trust that they were performing the image signature verification correctly, so I don't think enabling image signature verification removes the cloud infrastructure from the trust loop.

So why use the image signature verification at all?  It was implemented during the Liberty-Mitaka timeframe, when glance was still using md5 for checksumming.  The first md5 collision was produced sometime around 2004, but md5 was still thought circa 2015 to be OK as a quick check that no bits were flipped during data transfer.  Image signature verification was implemented *instead* of using a stronger checksumming mechanism in glance to provide data verification, but since it required the deployment and maintenance of a key management service, as well as requiring extra work by the end user to compute the signature and add the appropriate image properties to the image, it didn't catch on, and the secure 'multihash' feature, which doesn't require key management and is as automatic as the old md5 checksum, was added to glance in Rocky.

In my opinion, having signature validation disabled doesn't affect the security of image data in a Rocky or later OpenStack cloud.  The only issue I can see is if for some reason a workflow requires using a signed image (not because it's better, but because that's what a customer wants).  In that case, verify_glance_signatures can be enabled, and the other workaround comes into play, namely, removing the 'signature_verified' metadata from all images in glance.

Comment 23 Luigi Toscano 2023-06-12 14:52:53 UTC
*** Bug 2121573 has been marked as a duplicate of this bug. ***