Bug 2139031 - Pods that are using ocs-storagecluster-cephfs StorageClass failing with Error: failed to resolve symlink "/var/lib/kubelet/pods" permission denied [NEEDINFO]
Summary: Pods that are using ocs-storagecluster-cephfs StorageClass failing with Error...
Keywords:
Status: NEW
Alias: None
Product: Red Hat OpenShift Container Storage
Classification: Red Hat Storage
Component: ceph
Version: unspecified
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: ---
: ---
Assignee: Scott Ostapovicz
QA Contact: Neha Berry
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-11-01 03:36 UTC by Ashwini M. Khaire
Modified: 2023-08-03 08:30 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Embargoed:
akhaire: needinfo? (sostapov)
mrajanna: needinfo? (akhaire)

Attachments (Terms of Use)

Description Ashwini M. Khaire 2022-11-01 03:36:16 UTC 
Description of problem (please be detailed as possible and provide log
snippests):

Pods that are using storage from the `ocs-storagecluster-cephfs` StorageClass are stuck into CreateContainerError post restart and failing with permission denied errors on a specific node.

~~~
Warning  Failed                  5m11s (x12 over 7m16s)  kubelet                  Error: failed to resolve symlink "/var/lib/kubelet/pods/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx/volumes/kubernetes.io~csi/pvc-xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx/mount": lstat /var/lib/kubelet/pods/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx/volumes/kubernetes.io~csi/pvc-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx/mount: permission denied
~~~

Version of all relevant components (if applicable):

OpenShift Cluster Version: v4.10
OpenShift Container version: v4.10

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?

The issue is impacting CU's production workload.

Is there any workaround available to the best of your knowledge?

Re-scheduling the pod to another node seems to resolve the issue.

Actual results:

The pod is failing with the permission denied error while using the `ocs-storagecluster-cephfs` StorageClass.

Expected results:

The pod should be up and running fine.
Note You need to log in before you can comment on or make changes to this bug.