Description of problem: Setup: rhos 16.1.4 , Ceph backend for nova, glance and cinder. Trying to extend encrypted volume while attached fails. 2021-01-28 09:45:40.696 6 DEBUG nova.compute.manager [req-97c509cf-9857-4c92-91c3-952ef3282834 31c90732f6c8492188623aa2d6548d63 3e004ad2953a4aa7a2f9022be3ffc7cd - default default] [instance: 8d640d15-30dd-4e72-a9ba-d9f7cf11b1ec] Received event volume-extended-d721825d-038a-42f6-8127-aaec171e5c39 external_instance_event /usr/lib/python3.6/site-packages/nova/compute/manager.py:9310 2021-01-28 09:45:40.718 6 DEBUG nova.compute.manager [req-97c509cf-9857-4c92-91c3-952ef3282834 31c90732f6c8492188623aa2d6548d63 3e004ad2953a4aa7a2f9022be3ffc7cd - default default] [instance: 8d640d15-30dd-4e72-a9ba-d9f7cf11b1ec] Handling volume-extended event for volume d721825d-038a-42f6-8127-aaec171e5c39 extend_volume /usr/lib/python3.6/site-packages/nova/compute/manager.py:9157 2021-01-28 09:45:40.740 6 INFO nova.compute.manager [req-97c509cf-9857-4c92-91c3-952ef3282834 31c90732f6c8492188623aa2d6548d63 3e004ad2953a4aa7a2f9022be3ffc7cd - default default] [instance: 8d640d15-30dd-4e72-a9ba-d9f7cf11b1ec] Cinder extended volume d721825d-038a-42f6-8127-aaec171e5c39; extending it to detect new size 2021-01-28 09:45:40.993 6 DEBUG os_brick.encryptors [req-97c509cf-9857-4c92-91c3-952ef3282834 31c90732f6c8492188623aa2d6548d63 3e004ad2953a4aa7a2f9022be3ffc7cd - default default] Using volume encryption metadata '{'encryption_key_id': '***', 'control_location': 'front-end', 'cipher': 'aes-xts-plain64', 'key_size': 256, 'provider': 'luks'}' for connection: {'driver_volume_type': 'rbd', 'data': {'name': 'volumes/volume-d721825d-038a-42f6-8127-aaec171e5c39', 'hosts': ['172.17.3.139'], 'ports': ['6789'], 'cluster_name': 'ceph', 'auth_enabled': True, 'auth_username': 'openstack', 'secret_type': 'ceph', 'secret_uuid': '***', 'volume_id': 'd721825d-038a-42f6-8127-aaec171e5c39', 'discard': True, 'keyring': None, 'qos_specs': None, 'access_mode': 'rw', 'encrypted': True}, 'status': 'reserved', 'instance': '8d640d15-30dd-4e72-a9ba-d9f7cf11b1ec', 'attached_at': '', 'detached_at': '', 'volume_id': 'd721825d-038a-42f6-8127-aaec171e5c39', 'serial': 'd721825d-038a-42f6-8127-aaec171e5c39'} get_encryption_metadata /usr/lib/python3.6/site-packages/os_brick/encryptors/__init__.py:127 2021-01-28 09:45:41.142 6 ERROR nova.virt.libvirt.driver [req-97c509cf-9857-4c92-91c3-952ef3282834 31c90732f6c8492188623aa2d6548d63 3e004ad2953a4aa7a2f9022be3ffc7cd - default default] [instance: 8d640d15-30dd-4e72-a9ba-d9f7cf11b1ec] Unknown error when attempting to find the payload_offset for LUKSv1 encrypted disk rbd:volumes/volume-d721825d-038a-42f6-8127-aaec171e5c39.: nova.exception.InvalidDiskInfo: Disk info file is invalid: qemu-img failed to execute on rbd:volumes/volume-d721825d-038a-42f6-8127-aaec171e5c39 : Unexpected error while running command. Command: /usr/libexec/platform-python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info rbd:volumes/volume-d721825d-038a-42f6-8127-aaec171e5c39 --output=json --force-share Exit code: 1 Stdout: '' Stdout: '' Stderr: "qemu-img: Could not open 'rbd:volumes/volume-d721825d-038a-42f6-8127-aaec171e5c39': error connecting: Permission denied\n" 2021-01-28 09:45:41.142 6 ERROR nova.virt.libvirt.driver [instance: 8d640d15-30dd-4e72-a9ba-d9f7cf11b1ec] Traceback (most recent call last): 2021-01-28 09:45:41.142 6 ERROR nova.virt.libvirt.driver [instance: 8d640d15-30dd-4e72-a9ba-d9f7cf11b1ec] File "/usr/lib/python3.6/site-packages/nova/virt/libvirt/driver.py", line 2156, in _resize_attached_encrypted_volume 2021-01-28 09:45:41.142 6 ERROR nova.virt.libvirt.driver [instance: 8d640d15-30dd-4e72-a9ba-d9f7cf11b1ec] qemu_version=self._host.get_connection().getVersion()) 2021-01-28 09:45:41.142 6 ERROR nova.virt.libvirt.driver [instance: 8d640d15-30dd-4e72-a9ba-d9f7cf11b1ec] File "/usr/lib/python3.6/site-packages/oslo_privsep/priv_context.py", line 245, in _wrap 2021-01-28 09:45:41.142 6 ERROR nova.virt.libvirt.driver [instance: 8d640d15-30dd-4e72-a9ba-d9f7cf11b1ec] return self.channel.remote_call(name, args, kwargs) 2021-01-28 09:45:41.142 6 ERROR nova.virt.libvirt.driver [instance: 8d640d15-30dd-4e72-a9ba-d9f7cf11b1ec] File "/usr/lib/python3.6/site-packages/oslo_privsep/daemon.py", line 224, in remote_call 2021-01-28 09:45:41.142 6 ERROR nova.virt.libvirt.driver [instance: 8d640d15-30dd-4e72-a9ba-d9f7cf11b1ec] raise exc_type(*result[2]) 2021-01-28 09:45:41.142 6 ERROR nova.virt.libvirt.driver [instance: 8d640d15-30dd-4e72-a9ba-d9f7cf11b1ec] nova.exception.InvalidDiskInfo: Disk info file is invalid: qemu-img failed to execute on rbd:volumes/volume-d721825d-038a-42f6-8127-aaec171e5c39 : Unexpected error while running command. 2021-01-28 09:45:41.142 6 ERROR nova.virt.libvirt.driver [instance: 8d640d15-30dd-4e72-a9ba-d9f7cf11b1ec] Command: /usr/libexec/platform-python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info rbd:volumes/volume-d721825d-038a-42f6-8127-aaec171e5c39 --output=json --force-share 2021-01-28 09:45:41.142 6 ERROR nova.virt.libvirt.driver [instance: 8d640d15-30dd-4e72-a9ba-d9f7cf11b1ec] Exit code: 1 2021-01-28 09:45:41.142 6 ERROR nova.virt.libvirt.driver [instance: 8d640d15-30dd-4e72-a9ba-d9f7cf11b1ec] Stdout: '' 2021-01-28 09:45:41.142 6 ERROR nova.virt.libvirt.driver [instance: 8d640d15-30dd-4e72-a9ba-d9f7cf11b1ec] Stderr: "qemu-img: Could not open 'rbd:volumes/volume-d721825d-038a-42f6-8127-aaec171e5c39': error connecting: Permission denied\n" 2021-01-28 09:45:41.142 6 ERROR nova.virt.libvirt.driver [instance: 8d640d15-30dd-4e72-a9ba-d9f7cf11b1ec] 2021-01-28 09:45:41.144 6 WARNING nova.compute.manager [req-97c509cf-9857-4c92-91c3-952ef3282834 31c90732f6c8492188623aa2d6548d63 3e004ad2953a4aa7a2f9022be3ffc7cd - default default] [instance: 8d640d15-30dd-4e72-a9ba-d9f7cf11b1ec] Extend volume failed, volume_id=d721825d-038a-42f6-8127-aaec171e5c39, reason: Disk info file is invalid: qemu-img failed to execute on rbd:volumes/volume-d721825d-038a-42f6-8127-aaec171e5c39 : Unexpected error while running command. Command: /usr/libexec/platform-python -m oslo_concurrency.prlimit --as=1073741824 --cpu=30 -- env LC_ALL=C LANG=C qemu-img info rbd:volumes/volume-d721825d-038a-42f6-8127-aaec171e5c39 --output=json --force-share Exit code: 1 Stdout: '' Stderr: "qemu-img: Could not open 'rbd:volumes/volume-d721825d-038a-42f6-8127-aaec171e5c39': error connecting: Permission denied\n": nova.exception.InvalidDiskInfo: Disk info file is invalid: qemu-img failed to execute on rbd:volumes/volume-d721825d-038a-42f6-8127-aaec171e5c39 : Unexpected error while running command.
Created attachment 1751633 [details] nova_compute.log
Yeah this is valid, the code I wrote assumed qemu-img had access to the admin client keyring that isn't the case in OSP within the nova_compute container so we need to use the connection_info to point at the correct keyring. I'll get this fixed shortly.
Just to illustrate the issue here: $ podman exec -ti -u root nova_compute bash $ qemu-img info rbd:volumes/volume-89e34e6b-3e7c-402e-935f-1e2c7f007d6c qemu-img: Could not open 'rbd:volumes/volume-89e34e6b-3e7c-402e-935f-1e2c7f007d6c': error connecting: Permission denied $ qemu-img info rbd:volumes/volume-89e34e6b-3e7c-402e-935f-1e2c7f007d6c:id=openstack image: json:{"driver": "raw", "file": {"pool": "volumes", "image": "volume-89e34e6b-3e7c-402e-935f-1e2c7f007d6c", "driver": "rbd", "user": "openstack"}} file format: raw virtual size: 1 GiB (1073741824 bytes) disk size: unavailable cluster_size: 4194304
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Release of components for Red Hat OpenStack Platform 17.0 (Wallaby)), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2022:6543