Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2168966

Summary: [RHOS-17.1] Permission denied when attempting to attach an encrypted luks volume
Product: Red Hat OpenStack Reporter: James Parker <jparker>
Component: openstack-novaAssignee: OSP DFG:Compute <osp-dfg-compute>
Status: CLOSED CURRENTRELEASE QA Contact: OSP DFG:Compute <osp-dfg-compute>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 17.1 (Wallaby)CC: alifshit, dasmith, eglynn, jgrosso, jhakimra, kchamart, ltoscano, sbauza, sgordon, vromanso
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-03-17 14:02:52 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description James Parker 2023-02-10 18:41:37 UTC 
Created attachment 1943382 [details]
Combine logs around failed attachment

Created attachment 1943382 [details]
Combine logs around failed attachment

Description of problem:  This is triggered in phase3 CI via testcase [1].  Test is creating a server and two volumes.  First volume is unencrypted and the second is encrypted.  Unencrypted volume attaches without any issue but the follow up attachment of the encrypted volume fails.  Checking the logs I see the following:


  2023-02-04 00:40:16.080+0000: 22524: debug : qemuDomainObjEnterMonitorInternal:6077 : Entering monitor (mon=0x7f42e800c5d0 vm=0x7f42f40183d0 name=instance-00000028)
2023-02-04 00:40:16.080+0000: 22524: debug : qemuMonitorBlockdevAdd:3942 : props=0x7f42f8049e40 (node-name=libvirt-4-storage)
2023-02-04 00:40:16.080+0000: 22524: debug : qemuMonitorBlockdevAdd:3945 : mon:0x7f42e800c5d0 vm:0x7f42f40183d0 fd:30
2023-02-04 00:40:16.081+0000: 22524: info : qemuMonitorSend:861 : QEMU_MONITOR_SEND_MSG: mon=0x7f42e800c5d0 msg={"execute":"blockdev-add","arguments":{"driver":"host_device","filename":"/dev/disk/by-id/os-brick+dev+sdb","aio":"native","node-name":"libvirt-4-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"},"id":"libvirt-420"}
 fd=-1
2023-02-04 00:40:16.081+0000: 49438: info : qemuMonitorIOWrite:366 : QEMU_MONITOR_IO_WRITE: mon=0x7f42e800c5d0 buf={"execute":"blockdev-add","arguments":{"driver":"host_device","filename":"/dev/disk/by-id/os-brick+dev+sdb","aio":"native","node-name":"libvirt-4-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"},"id":"libvirt-420"}
 len=258 ret=258 errno=0
2023-02-04 00:40:16.083+0000: 49438: debug : qemuMonitorJSONIOProcessLine:189 : Line [{"id": "libvirt-420", "error": {"class": "GenericError", "desc": "Could not open '/dev/disk/by-id/os-brick+dev+sdb': Permission denied"}}]
2023-02-04 00:40:16.083+0000: 49438: info : qemuMonitorJSONIOProcessLine:208 : QEMU_MONITOR_RECV_REPLY: mon=0x7f42e800c5d0 reply={"id": "libvirt-420", "error": {"class": "GenericError", "desc": "Could not open '/dev/disk/by-id/os-brick+dev+sdb': Permission denied"}}
2023-02-04 00:40:16.083+0000: 22524: debug : qemuMonitorJSONCheckErrorFull:354 : unable to execute QEMU command {"execute":"blockdev-add","arguments":{"driver":"host_device","filename":"/dev/disk/by-id/os-brick+dev+sdb","aio":"native","node-name":"libvirt-4-storage","cache":{"direct":true,"no-flush":false},"auto-read-only":true,"discard":"unmap"},"id":"libvirt-420"}: {"id":"libvirt-420","error":{"class":"GenericError","desc":"Could not open '/dev/disk/by-id/os-brick+dev+sdb': Permission denied"}}
2023-02-04 00:40:16.083+0000: 22524: error : qemuMonitorJSONCheckErrorFull:366 : internal error: unable to execute QEMU command 'blockdev-add': Could not open '/dev/disk/by-id/os-brick+dev+sdb': Permission denied
2023-02-04 00:40:16.083+0000: 22524: debug : qemuDomainObjExitMonitor:6106 : Exited monitor (mon=0x7f42e800c5d0 vm=0x7f42f40183d0 name=instance-00000028)
2023-02-04 00:40:16.083+0000: 22524: debug : qemuDomainObjEnterMonitorInternal:6077 : Entering monitor (mon=0x7f42e800c5d0 vm=0x7f42f40183d0 name=instance-00000028)
2023-02-04 00:40:16.083+0000: 22524: debug : qemuDomainObjExitMonitor:6106 : Exited monitor (mon=0x7f42e800c5d0 vm=0x7f42f40183d0 name=instance-00000028)
2023-02-04 00:40:16.083+0000: 22524: debug : qemuDomainStorageSourceAccessModify:7919 : src='/dev/disk/by-id/os-brick+dev+sdb' readonly=0 force_ro=0 force_rw=0 revoke=1 chain=1
2023-02-04 00:40:16.083+0000: 22524: debug : qemuCgroupDenyDevicePath:112 : Deny path /dev/mapper/control, perms: rwm
2023-02-04 00:40:16.083+0000: 22524: debug : virCgroupV2DenyDevice:1853 : nothing to do, device is not allowed
2023-02-04 00:40:16.083+0000: 22524: debug : qemuTeardownImageCgroup:294 : Deny path /dev/disk/by-id/os-brick+dev+sdb
2023-02-04 00:40:16.083+0000: 22524: debug : qemuCgroupDenyDevicePath:112 : Deny path /dev/disk/by-id/os-brick+dev+sdb, perms: rwm
2023-02-04 00:40:16.089+0000: 22524: debug : virThreadJobClear:118 : Thread 22524 (rpc-virtqemud) finished job remoteDispatchDomainAttachDeviceFlags with ret=-1
2023-02-04 00:40:16.090 2 ERROR nova.virt.libvirt.driver [req-a2afc2d9-5c49-444b-b919-11e8c43d1598 f2e24fbef1974222979c05491e6119d5 19b534db0e164e988dcc6aad92f74042 - default default] [instance: b0603572-dcea-4d10-965c-28e0aebc2e55] Failed to attach volume at mountpoint: /dev/vdc: libvirt.libvirtError: internal error: unable to execute QEMU command 'blockdev-add': Could not open '/dev/disk/by-id/os-brick+dev+sdb': Permission denied
2023-02-04 00:40:16.090 2 ERROR nova.virt.libvirt.driver [instance: b0603572-dcea-4d10-965c-28e0aebc2e55] Traceback (most recent call last):
2023-02-04 00:40:16.090 2 ERROR nova.virt.libvirt.driver [instance: b0603572-dcea-4d10-965c-28e0aebc2e55]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/driver.py", line 2159, in attach_volume
2023-02-04 00:40:16.090 2 ERROR nova.virt.libvirt.driver [instance: b0603572-dcea-4d10-965c-28e0aebc2e55]     guest.attach_device(conf, persistent=True, live=live)
2023-02-04 00:40:16.090 2 ERROR nova.virt.libvirt.driver [instance: b0603572-dcea-4d10-965c-28e0aebc2e55]   File "/usr/lib/python3.9/site-packages/nova/virt/libvirt/guest.py", line 321, in attach_device
2023-02-04 00:40:16.090 2 ERROR nova.virt.libvirt.driver [instance: b0603572-dcea-4d10-965c-28e0aebc2e55]     self._domain.attachDeviceFlags(device_xml, flags=flags)
2023-02-04 00:40:16.090 2 ERROR nova.virt.libvirt.driver [instance: b0603572-dcea-4d10-965c-28e0aebc2e55]   File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 190, in doit
2023-02-04 00:40:16.090 2 ERROR nova.virt.libvirt.driver [instance: b0603572-dcea-4d10-965c-28e0aebc2e55]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2023-02-04 00:40:16.090 2 ERROR nova.virt.libvirt.driver [instance: b0603572-dcea-4d10-965c-28e0aebc2e55]   File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 148, in proxy_call
2023-02-04 00:40:16.090 2 ERROR nova.virt.libvirt.driver [instance: b0603572-dcea-4d10-965c-28e0aebc2e55]     rv = execute(f, *args, **kwargs)
2023-02-04 00:40:16.090 2 ERROR nova.virt.libvirt.driver [instance: b0603572-dcea-4d10-965c-28e0aebc2e55]   File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 129, in execute
2023-02-04 00:40:16.090 2 ERROR nova.virt.libvirt.driver [instance: b0603572-dcea-4d10-965c-28e0aebc2e55]     six.reraise(c, e, tb)
2023-02-04 00:40:16.090 2 ERROR nova.virt.libvirt.driver [instance: b0603572-dcea-4d10-965c-28e0aebc2e55]   File "/usr/lib/python3.9/site-packages/six.py", line 709, in reraise
2023-02-04 00:40:16.090 2 ERROR nova.virt.libvirt.driver [instance: b0603572-dcea-4d10-965c-28e0aebc2e55]     raise value
2023-02-04 00:40:16.090 2 ERROR nova.virt.libvirt.driver [instance: b0603572-dcea-4d10-965c-28e0aebc2e55]   File "/usr/lib/python3.9/site-packages/eventlet/tpool.py", line 83, in tworker
2023-02-04 00:40:16.090 2 ERROR nova.virt.libvirt.driver [instance: b0603572-dcea-4d10-965c-28e0aebc2e55]     rv = meth(*args, **kwargs)
2023-02-04 00:40:16.090 2 ERROR nova.virt.libvirt.driver [instance: b0603572-dcea-4d10-965c-28e0aebc2e55]   File "/usr/lib64/python3.9/site-packages/libvirt.py", line 716, in attachDeviceFlags
2023-02-04 00:40:16.090 2 ERROR nova.virt.libvirt.driver [instance: b0603572-dcea-4d10-965c-28e0aebc2e55]     raise libvirtError('virDomainAttachDeviceFlags() failed')
2023-02-04 00:40:16.090 2 ERROR nova.virt.libvirt.driver [instance: b0603572-dcea-4d10-965c-28e0aebc2e55] libvirt.libvirtError: internal error: unable to execute QEMU command 'blockdev-add': Could not open '/dev/disk/by-id/os-brick+dev+sdb': Permission denied
2023-02-04 00:40:16.090 2 ERROR nova.virt.libvirt.driver [instance: b0603572-dcea-4d10-965c-28e0aebc2e55] 
2023-02-04 00:40:16.098+0000: 22527: debug : virThreadJobSet:93 : Thread 22527 (rpc-virtqemud) is now running job remoteDispatchConnectGetLibVersion
2023-02-04 00:40:16.098+0000: 22527: debug : virConnectGetLibVersion:234 : conn=0x7f42ec0125b0, libVir=0x7f42feffc8e0
2023-02-04 00:40:16.098+0000: 22527: debug : virThreadJobClear:118 : Thread 22527 (rpc-virtqemud) finished job remoteDispatchCon


Version-Release number of selected component (if applicable):
RHOS-17.1-RHEL-9-20230131.n.2

How reproducible:
100%

Steps to Reproduce:
1. Create a server and an encrypted luks volume
2. Attach the volume to the server
3.

Actual results:
Encrypted volume fails to attach

Expected results:
Encrypted volume attaches correctly

Additional info:
Relevant combine logs attached, relevant id's for the logs below:
attach volume request uuid: req-a2afc2d9-5c49-444b-b919-11e8c43d1598
volume-uuid: 6bec9616-26c6-470c-bca2-20c0d221a031
server-uuid: b0603572-dcea-4d10-965c-28e0aebc2e55
Comment 9 Luigi Toscano 2023-03-10 21:35:35 UTC 
*** Bug 2175225 has been marked as a duplicate of this bug. ***