Bug 2083953
| Summary: | [Tracker for Ceph BZ #2084579] PVC created with ocs-storagecluster-ceph-nfs storageclass is moving to pending status | |||
|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat OpenShift Data Foundation | Reporter: | Amrita Mahapatra <ammahapa> | |
| Component: | ceph | Assignee: | Scott Ostapovicz <sostapov> | |
| ceph sub component: | RBD | QA Contact: | Amrita Mahapatra <ammahapa> | |
| Status: | CLOSED ERRATA | Docs Contact: | ||
| Severity: | high | |||
| Priority: | high | CC: | bniver, madam, muagarwa, ndevos, ocs-bugs, odf-bz-bot, pnataraj, rar | |
| Version: | 4.11 | Keywords: | Tracking | |
| Target Milestone: | --- | |||
| Target Release: | ODF 4.11.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | 4.11.0-85 | Doc Type: | No Doc Update | |
| Doc Text: | Story Points: | --- | ||
| Clone Of: | ||||
| : | 2084579 (view as bug list) | Environment: | ||
| Last Closed: | 2022-08-24 13:52:07 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | 2084579 | |||
| Bug Blocks: | ||||
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.11.0 security, enhancement, & bugfix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:6156 |
Description of problem (please be detailed as possible and provide log snippests): For ODF cluster with nfs enabled, pvc created with storageclass ocs-storagecluster-ceph-nfs is moving to pending status the event failed with error, Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Provisioning 34s (x8 over 100s) openshift-storage.nfs.csi.ceph.com_csi-nfsplugin-provisioner-f85b75fbb-2mfzg_8597901e-a5c7-4282-b3c0-4dfbe513af9b External provisioner is provisioning volume for claim "openshift-storage/nfs-pvc1" Warning ProvisioningFailed 33s (x8 over 99s) openshift-storage.nfs.csi.ceph.com_csi-nfsplugin-provisioner-f85b75fbb-2mfzg_8597901e-a5c7-4282-b3c0-4dfbe513af9b failed to provision volume with StorageClass "ocs-storagecluster-ceph-nfs": rpc error: code = InvalidArgument desc = failed to create export: failed to create export "0001-0011-openshift-storage-0000000000000001-2798e7c2-d075-11ec-948f-0a580a83001d" in NFS-cluster "ocs-storagecluster-cephnfs"(an error (exit status 22) occurred while running ceph args: [--id csi-cephfs-provisioner --keyfile=***stripped*** -m 172.30.126.91:6789,172.30.89.5:6789,172.30.223.225:6789 nfs export create cephfs ocs-storagecluster-cephfilesystem ocs-storagecluster-cephnfs /0001-0011-openshift-storage-0000000000000001-2798e7c2-d075-11ec-948f-0a580a83001d /volumes/csi/nfs-export-2798e7c2-d075-11ec-948f-0a580a83001d/60a0601a-f340-48b3-9638-05ed83c80f11]): Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1386, in _handle_command return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 397, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/nfs/module.py", line 42, in _cmd_nfs_export_create_cephfs squash=squash, addr=client_addr) File "/usr/share/ceph/mgr/nfs/export.py", line 33, in cluster_check if kwargs['cluster_id'] not in available_clusters(export.mgr): File "/usr/share/ceph/mgr/nfs/utils.py", line 19, in available_clusters completion = mgr.describe_service(service_type='nfs') File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1387, in inner completion = self._oremote(method_name, args, kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1453, in _oremote raise NoOrchestrator() orchestrator._interface.NoOrchestrator: No orchestrator configured (try `ceph orch set backend`) Normal ExternalProvisioning 3s (x9 over 100s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "openshift-storage.nfs.csi.ceph.com" or manually created by system administrator Version of all relevant components (if applicable): OCP: 4.11.0-0.nightly-2022-05-09-224745 ODF full version: 4.11.0-67 Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? yes Is there any workaround available to the best of your knowledge? yes Run below ceph commands inside rook ceph toolbox. ceph mgr module enable rook ceph mgr module enable nfs ceph orch set backend rook Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 3 Can this issue reproducible? yes Can this issue reproduce from the UI? To enable nfs need to use CLI, pvc can be created from UI after nfs ganesha server is up and nfs provisioner pods are running. If this is a regression, please provide more details to justify this: No Steps to Reproduce: 1. Update nfs feature for the storage-cluster with patch command. [ammahapa@ammahapa ~]$ oc patch -n openshift-storage storageclusters.ocs.openshift.io ocs- storagecluster --patch '{"spec": {"nfs":{"enable": true}}}' --type merge storagecluster.ocs.openshift.io/ocs-storagecluster patched 2. Enable "ROOK_CSI_ENABLE_NFS" [ammahapa@ammahapa ~]$ oc patch cm rook-ceph-operator-config -n openshift-storage -p $'data:\n "ROOK_CSI_ENABLE_NFS": "true"' configmap/rook-ceph-operator-config patched 3. Create pvc with ocs-storagecluster-ceph-nfs storage class Actual results: Created pvc is in Pending status, failing with error, Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Provisioning 34s (x8 over 100s) openshift-storage.nfs.csi.ceph.com_csi-nfsplugin-provisioner-f85b75fbb-2mfzg_8597901e-a5c7-4282-b3c0-4dfbe513af9b External provisioner is provisioning volume for claim "openshift-storage/nfs-pvc1" Warning ProvisioningFailed 33s (x8 over 99s) openshift-storage.nfs.csi.ceph.com_csi-nfsplugin-provisioner-f85b75fbb-2mfzg_8597901e-a5c7-4282-b3c0-4dfbe513af9b failed to provision volume with StorageClass "ocs-storagecluster-ceph-nfs": rpc error: code = InvalidArgument desc = failed to create export: failed to create export "0001-0011-openshift-storage-0000000000000001-2798e7c2-d075-11ec-948f-0a580a83001d" in NFS-cluster "ocs-storagecluster-cephnfs"(an error (exit status 22) occurred while running ceph args: [--id csi-cephfs-provisioner --keyfile=***stripped*** -m 172.30.126.91:6789,172.30.89.5:6789,172.30.223.225:6789 nfs export create cephfs ocs-storagecluster-cephfilesystem ocs-storagecluster-cephnfs /0001-0011-openshift-storage-0000000000000001-2798e7c2-d075-11ec-948f-0a580a83001d /volumes/csi/nfs-export-2798e7c2-d075-11ec-948f-0a580a83001d/60a0601a-f340-48b3-9638-05ed83c80f11]): Error EINVAL: Traceback (most recent call last): File "/usr/share/ceph/mgr/mgr_module.py", line 1386, in _handle_command return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf) File "/usr/share/ceph/mgr/mgr_module.py", line 397, in call return self.func(mgr, **kwargs) File "/usr/share/ceph/mgr/nfs/module.py", line 42, in _cmd_nfs_export_create_cephfs squash=squash, addr=client_addr) File "/usr/share/ceph/mgr/nfs/export.py", line 33, in cluster_check if kwargs['cluster_id'] not in available_clusters(export.mgr): File "/usr/share/ceph/mgr/nfs/utils.py", line 19, in available_clusters completion = mgr.describe_service(service_type='nfs') File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1387, in inner completion = self._oremote(method_name, args, kwargs) File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 1453, in _oremote raise NoOrchestrator() orchestrator._interface.NoOrchestrator: No orchestrator configured (try `ceph orch set backend`) Normal ExternalProvisioning 3s (x9 over 100s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "openshift-storage.nfs.csi.ceph.com" or manually created by system administrator Expected results: Created pvc should move to Bound status.