Created attachment 1957836 [details] attached the testcase of the failure Description of problem (please be detailed as possible and provide log snippests): Testcase was failing with the below error : """ > raise ResourceWrongStatusException(resource.name, resource.describe()) E ocs_ci.ocs.exceptions.ResourceWrongStatusException: Resource pod-test-cephfs-7587b15391e24385aa2b6df2 describe output: Name: pod-test-cephfs-7587b15391e24385aa2b6df2 E Namespace: namespace-test-aef02c6fd71b40ba94da79a82 E Priority: 0 E Service Account: default E Node: sao01-worker-1.rdr-abhinew.ibm.com/192.168.0.83 E Start Time: Wed, 05 Apr 2023 02:08:35 -0400 E Labels: <none> E Annotations: k8s.ovn.org/pod-networks: E {"default":{"ip_addresses":["10.129.3.30/23"],"mac_address":"0a:58:0a:81:03:1e","gateway_ips":["10.129.2.1"],"ip_address":"10.129.3.30/23"... E openshift.io/scc: privileged E Status: Pending E IP: E IPs: <none> E Containers: E web-server: E Container ID: E Image: quay.io/ocsci/nginx:latest E Image ID: E Port: <none> E Host Port: <none> E State: Waiting E Reason: ContainerCreating E Ready: False E Restart Count: 0 E Environment: <none> E Mounts: E /var/lib/www/html from mypvc (rw) E /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-djcbb (ro) E Conditions: E Type Status E Initialized True E Ready False E ContainersReady False E PodScheduled True E Volumes: E mypvc: E Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) E ClaimName: pvc-test-4d92c28aac684d3db4f1251f9df45a8 E ReadOnly: false E kube-api-access-djcbb: E Type: Projected (a volume that contains injected data from multiple sources) E TokenExpirationSeconds: 3607 E ConfigMapName: kube-root-ca.crt E ConfigMapOptional: <nil> E DownwardAPI: true E ConfigMapName: openshift-service-ca.crt E ConfigMapOptional: <nil> E QoS Class: BestEffort E Node-Selectors: <none> E Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s E node.kubernetes.io/unreachable:NoExecute op=Exists for 300s E Events: E Type Reason Age From Message E ---- ------ ---- ---- ------- E Warning FailedMount 56s (x2 over 2m59s) kubelet Unable to attach or mount volumes: unmounted volumes=[mypvc], unattached volumes=[mypvc kube-api-access-djcbb]: timed out waiting for the condition E Warning FailedMount 52s (x10 over 5m2s) kubelet MountVolume.SetUp failed for volume "pvc-a03d9690-9a87-4cf2-a73f-5b5fd65b196a" : rpc error: code = Internal desc = stat /var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.cephfs.csi.ceph.com/cafb8df0afa1133d486552983ffa9d43376ea73810f04ceb3ea40f58f5a71c1e/globalmount: permission denied ocs_ci/helpers/helpers.py:122: ResourceWrongStatusException Version of all relevant components (if applicable): ODF version : 4.13 OCS cluster version : 4.13 Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? Is there any workaround available to the best of your knowledge? Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? Can this issue reproducible? Can this issue reproduce from the UI? If this is a regression, please provide more details to justify this: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Pod is not coming up because of this reason: Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedInterface 100s multus Add eth0 [10.128.2.141/23] from ovn-kubernetes Normal Pulled 98s kubelet Successfully pulled image "quay.io/ocsci/nginx:latest" in 2.151563841s (2.151575966s including waiting) Normal Pulled 95s kubelet Successfully pulled image "quay.io/ocsci/nginx:latest" in 2.203875332s (2.203889245s including waiting) Normal Pulled 79s kubelet Successfully pulled image "quay.io/ocsci/nginx:latest" in 2.199828444s (2.199842167s including waiting) Normal Pulled 64s kubelet Successfully pulled image "quay.io/ocsci/nginx:latest" in 2.187749438s (2.187763312s including waiting) Normal Pulled 47s kubelet Successfully pulled image "quay.io/ocsci/nginx:latest" in 2.099495679s (2.0995146s including waiting) Normal Pulled 32s kubelet Successfully pulled image "quay.io/ocsci/nginx:latest" in 2.164546863s (2.164561013s including waiting) Warning Failed 16s (x7 over 98s) kubelet Error: failed to resolve symlink "/var/lib/kubelet/pods/1dc77717-b3a0-424c-a249-6a4ba29052e8/volumes/kubernetes.io~csi/pvc-75aa0237-ef71-456b-a04c-cd8485bcf472/mount": lstat /var/lib/kubelet/pods/1dc77717-b3a0-424c-a249-6a4ba29052e8/volumes/kubernetes.io~csi/pvc-75aa0237-ef71-456b-a04c-cd8485bcf472/mount: permission denied Normal Pulled 16s kubelet Successfully pulled image "quay.io/ocsci/nginx:latest" in 2.240981686s (2.240997379s including waiting) Normal Pulling 2s (x8 over 100s) kubelet Pulling image "quay.io/ocsci/nginx:latest" In the worker node on which pod is scheduled : sh-5.1# ls -lrt /var/lib/kubelet/pods/1dc77717-b3a0-424c-a249-6a4ba29052e8/volumes/kubernetes.io~csi/ total 0 drwxr-x---. 3 root root 40 Apr 12 07:29 pvc-75aa0237-ef71-456b-a04c-cd8485bcf472 sh-5.1# sh-5.1# ls -lrt /var/lib/kubelet/pods/1dc77717-b3a0-424c-a249-6a4ba29052e8/volumes/kubernetes.io~csi/pvc-75aa0237-ef71-456b-a04c-cd8485bcf472/ ls: cannot access '/var/lib/kubelet/pods/1dc77717-b3a0-424c-a249-6a4ba29052e8/volumes/kubernetes.io~csi/pvc-75aa0237-ef71-456b-a04c-cd8485bcf472/mount': Permission denied total 4 d?????????? ? ? ? ? ? mount -rw-r--r--. 1 root root 378 Apr 12 07:29 vol_data.json
Milind, PTAL. I think there was another BZ similar to this related to the mount path being a symbolic link. I'll try to dig that up. Feel free to include Madhu from CSI team for details.
Hi Abhishek, Do we have the must-gather for this BZ and what is the exact ODF version used for testing?
Link for the must-gather file : https://drive.google.com/file/d/1tFsEiWNvjXAxzPwtM8-GFyP3mwqm4DrI/view?usp=sharing ODF version : full_version=4.13.0-121