Bug 2111424

Summary: Cephfs PVC creation fails on a FIPS enabled cluster with clusterwide encryption
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Rachael <rgeorge>
Component: cephAssignee: Kotresh HR <khiremat>
ceph sub component: CephFS QA Contact: Rachael <rgeorge>
Status: CLOSED CURRENTRELEASE Docs Contact:
Severity: high    
Priority: unspecified CC: bniver, dahorak, hyelloji, khiremat, mbukatov, mmuench, muagarwa, odf-bz-bot
Version: 4.11Keywords: Regression
Target Milestone: ---   
Target Release: ODF 4.11.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 4.11.0-130 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 2112101 (view as bug list) Environment:
Last Closed: 2024-04-05 17:02:33 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2112101    
Bug Blocks:    

Description Rachael 2022-07-27 09:56:35 UTC
Description of problem (please be detailed as possible and provide log
snippets):

On a FIPS enabled ODF cluster with clusterwide encryption, the creation of cephFS PVCs fails with the following error: 

$ oc describe pvc cephfs-1
Name:          cephfs-1
Namespace:     default
StorageClass:  ocs-storagecluster-cephfs
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: openshift-storage.cephfs.csi.ceph.com
               volume.kubernetes.io/storage-provisioner: openshift-storage.cephfs.csi.ceph.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type     Reason                Age                   From                                                                                                                      Message
  ----     ------                ----                  ----                                                                                                                      -------
  Normal   Provisioning          80s (x10 over 5m36s)  openshift-storage.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-757d576d77-jls2t_c68f4f63-98c1-4caa-a91b-7e51d9aeb82a  External provisioner is provisioning volume for claim "default/cephfs-1"
  Warning  ProvisioningFailed    80s (x10 over 5m36s)  openshift-storage.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-757d576d77-jls2t_c68f4f63-98c1-4caa-a91b-7e51d9aeb82a  failed to provision volume with StorageClass "ocs-storagecluster-cephfs": rpc error: code = Internal desc = rados: ret=-22, Invalid argument: "Traceback (most recent call last):\n  File \"/usr/share/ceph/mgr/mgr_module.py\", line 1446, in _handle_command\n    return self.handle_command(inbuf, cmd)\n  File \"/usr/share/ceph/mgr/volumes/module.py\", line 437, in handle_command\n    return handler(inbuf, cmd)\n  File \"/usr/share/ceph/mgr/volumes/module.py\", line 34, in wrap\n    return f(self, inbuf, cmd)\n  File \"/usr/share/ceph/mgr/volumes/module.py\", line 491, in _cmd_fs_subvolume_create\n    namespace_isolated=cmd.get('namespace_isolated', False))\n  File \"/usr/share/ceph/mgr/volumes/fs/volume.py\", line 171, in create_subvolume\n    with open_subvol(self.mgr, fs_handle, self.volspec, group, subvolname, SubvolumeOpType.CREATE) as subvolume:\n  File \"/usr/lib64/python3.6/contextlib.py\", line 81, in __enter__\n    return next(self.gen)\n  File \"/usr/share/ceph/mgr/volumes/fs/operations/subvolume.py\", line 72, in open_subvol\n    subvolume = loaded_subvolumes.get_subvolume_object(mgr, fs, vol_spec, group, subvolname)\n  File \"/usr/share/ceph/mgr/volumes/fs/operations/versions/__init__.py\", line 93, in get_subvolume_object\n    subvolume = SubvolumeBase(mgr, fs, vol_spec, group, subvolname)\n  File \"/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_base.py\", line 38, in __init__\n    self.load_config()\n  File \"/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_base.py\", line 129, in load_config\n    self.fs.stat(self.legacy_config_path)\n  File \"/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_base.py\", line 78, in legacy_config_path\n    m = md5()\nValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS\n"



Version of all relevant components (if applicable):
---------------------------------------------------
OCP: 4.11.0-0.nightly-2022-07-26-154822
ODF: odf-operator.v4.11.0            full_version=4.11.0-124


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
Yes, creation of cephfs fails on the cluster


Is there any workaround available to the best of your knowledge?
Not that I am aware of



Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
2


Can this issue reproducible?
Yes. The issue was also seen on clusters upgraded from ODF 4.10 to 4.11


Can this issue reproduce from the UI?
Yes

If this is a regression, please provide more details to justify this:
Yes, this error was not seen in previous runs



Steps to Reproduce:
-------------------
1. Deploy an ODF cluster with FIPS and clusterwide encryption enabled
2. Create a cephfs PVC



Actual results:
---------------

The cephfs PVC creation fails with the following error:

  Warning  ProvisioningFailed    80s (x10 over 5m36s)  openshift-storage.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-757d576d77-jls2t_c68f4f63-98c1-4caa-a91b-7e51d9aeb82a  failed to provision volume with StorageClass "ocs-storagecluster-cephfs": rpc error: code = Internal desc = rados: ret=-22, Invalid argument: "Traceback (most recent call last):\n  File \"/usr/share/ceph/mgr/mgr_module.py\", line 1446, in _handle_command\n    return self.handle_command(inbuf, cmd)\n  File \"/usr/share/ceph/mgr/volumes/module.py\", line 437, in handle_command\n    return handler(inbuf, cmd)\n  File \"/usr/share/ceph/mgr/volumes/module.py\", line 34, in wrap\n    return f(self, inbuf, cmd)\n  File \"/usr/share/ceph/mgr/volumes/module.py\", line 491, in _cmd_fs_subvolume_create\n    namespace_isolated=cmd.get('namespace_isolated', False))\n  File \"/usr/share/ceph/mgr/volumes/fs/volume.py\", line 171, in create_subvolume\n    with open_subvol(self.mgr, fs_handle, self.volspec, group, subvolname, SubvolumeOpType.CREATE) as subvolume:\n  File \"/usr/lib64/python3.6/contextlib.py\", line 81, in __enter__\n    return next(self.gen)\n  File \"/usr/share/ceph/mgr/volumes/fs/operations/subvolume.py\", line 72, in open_subvol\n    subvolume = loaded_subvolumes.get_subvolume_object(mgr, fs, vol_spec, group, subvolname)\n  File \"/usr/share/ceph/mgr/volumes/fs/operations/versions/__init__.py\", line 93, in get_subvolume_object\n    subvolume = SubvolumeBase(mgr, fs, vol_spec, group, subvolname)\n  File \"/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_base.py\", line 38, in __init__\n    self.load_config()\n  File \"/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_base.py\", line 129, in load_config\n    self.fs.stat(self.legacy_config_path)\n  File \"/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_base.py\", line 78, in legacy_config_path\n    m = md5()\nValueError: [digital envelope routines: EVP_DigestInit_ex] disabled for FIPS\n"


Expected results:
-----------------

PVC creation should be successful

Comment 8 Madhu Rajanna 2022-08-04 05:11:05 UTC
*** Bug 2115041 has been marked as a duplicate of this bug. ***