Bug 2187305 - MountVolume.SetUp failed for volume while creating pvc for IBM POWER in ODF 4.13 [NEEDINFO]
Summary: MountVolume.SetUp failed for volume while creating pvc for IBM POWER in ODF 4.13
Keywords:
Status: CLOSED INSUFFICIENT_DATA
Alias: None
Product: Red Hat OpenShift Data Foundation
Classification: Red Hat Storage
Component: ceph
Version: 4.13
Hardware: ppc64le
OS: Linux
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Milind Changire
QA Contact: Elad
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-04-17 12:18 UTC by Abhishek Mundada
Modified: 2023-08-09 16:37 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-08-08 13:05:52 UTC
Embargoed:
gfarnum: needinfo? (mchangir)
mchangir: needinfo? (amundada)


Attachments (Terms of Use)
attached the testcase of the failure (133.72 KB, text/plain)
2023-04-17 12:18 UTC, Abhishek Mundada
no flags Details

Description Abhishek Mundada 2023-04-17 12:18:34 UTC
Created attachment 1957836 [details]
attached the testcase of the failure

Description of problem (please be detailed as possible and provide log
snippests):
Testcase was failing with the below error :
"""
>           raise ResourceWrongStatusException(resource.name, resource.describe())
E           ocs_ci.ocs.exceptions.ResourceWrongStatusException: Resource pod-test-cephfs-7587b15391e24385aa2b6df2 describe output: Name:             pod-test-cephfs-7587b15391e24385aa2b6df2
E           Namespace:        namespace-test-aef02c6fd71b40ba94da79a82
E           Priority:         0
E           Service Account:  default
E           Node:             sao01-worker-1.rdr-abhinew.ibm.com/192.168.0.83
E           Start Time:       Wed, 05 Apr 2023 02:08:35 -0400
E           Labels:           <none>
E           Annotations:      k8s.ovn.org/pod-networks:
E                               {"default":{"ip_addresses":["10.129.3.30/23"],"mac_address":"0a:58:0a:81:03:1e","gateway_ips":["10.129.2.1"],"ip_address":"10.129.3.30/23"...
E                             openshift.io/scc: privileged
E           Status:           Pending
E           IP:
E           IPs:              <none>
E           Containers:
E             web-server:
E               Container ID:
E               Image:          quay.io/ocsci/nginx:latest
E               Image ID:
E               Port:           <none>
E               Host Port:      <none>
E               State:          Waiting
E                 Reason:       ContainerCreating
E               Ready:          False
E               Restart Count:  0
E               Environment:    <none>
E               Mounts:
E                 /var/lib/www/html from mypvc (rw)
E                 /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-djcbb (ro)
E           Conditions:
E             Type              Status
E             Initialized       True
E             Ready             False
E             ContainersReady   False
E             PodScheduled      True
E           Volumes:
E             mypvc:
E               Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
E               ClaimName:  pvc-test-4d92c28aac684d3db4f1251f9df45a8
E               ReadOnly:   false
E             kube-api-access-djcbb:
E               Type:                    Projected (a volume that contains injected data from multiple sources)
E               TokenExpirationSeconds:  3607
E               ConfigMapName:           kube-root-ca.crt
E               ConfigMapOptional:       <nil>
E               DownwardAPI:             true
E               ConfigMapName:           openshift-service-ca.crt
E               ConfigMapOptional:       <nil>
E           QoS Class:                   BestEffort
E           Node-Selectors:              <none>
E           Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
E                                        node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
E           Events:
E             Type     Reason       Age                  From     Message
E             ----     ------       ----                 ----     -------
E             Warning  FailedMount  56s (x2 over 2m59s)  kubelet  Unable to attach or mount volumes: unmounted volumes=[mypvc], unattached volumes=[mypvc kube-api-access-djcbb]: timed out waiting for the condition
E             Warning  FailedMount  52s (x10 over 5m2s)  kubelet  MountVolume.SetUp failed for volume "pvc-a03d9690-9a87-4cf2-a73f-5b5fd65b196a" : rpc error: code = Internal desc = stat /var/lib/kubelet/plugins/kubernetes.io/csi/openshift-storage.cephfs.csi.ceph.com/cafb8df0afa1133d486552983ffa9d43376ea73810f04ceb3ea40f58f5a71c1e/globalmount: permission denied

ocs_ci/helpers/helpers.py:122: ResourceWrongStatusException


Version of all relevant components (if applicable):
ODF version : 4.13
OCS cluster version : 4.13

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1.
2.
3.


Actual results:


Expected results:


Additional info:

Comment 2 Abhishek Mundada 2023-04-18 04:56:16 UTC
Pod is not coming up because of this reason:
Events:
  Type     Reason          Age                From     Message
  ----     ------          ----               ----     -------
  Normal   AddedInterface  100s               multus   Add eth0 [10.128.2.141/23] from ovn-kubernetes
  Normal   Pulled          98s                kubelet  Successfully pulled image "quay.io/ocsci/nginx:latest" in 2.151563841s (2.151575966s including waiting)
  Normal   Pulled          95s                kubelet  Successfully pulled image "quay.io/ocsci/nginx:latest" in 2.203875332s (2.203889245s including waiting)
  Normal   Pulled          79s                kubelet  Successfully pulled image "quay.io/ocsci/nginx:latest" in 2.199828444s (2.199842167s including waiting)
  Normal   Pulled          64s                kubelet  Successfully pulled image "quay.io/ocsci/nginx:latest" in 2.187749438s (2.187763312s including waiting)
  Normal   Pulled          47s                kubelet  Successfully pulled image "quay.io/ocsci/nginx:latest" in 2.099495679s (2.0995146s including waiting)
  Normal   Pulled          32s                kubelet  Successfully pulled image "quay.io/ocsci/nginx:latest" in 2.164546863s (2.164561013s including waiting)
  Warning  Failed          16s (x7 over 98s)  kubelet  Error: failed to resolve symlink "/var/lib/kubelet/pods/1dc77717-b3a0-424c-a249-6a4ba29052e8/volumes/kubernetes.io~csi/pvc-75aa0237-ef71-456b-a04c-cd8485bcf472/mount": lstat /var/lib/kubelet/pods/1dc77717-b3a0-424c-a249-6a4ba29052e8/volumes/kubernetes.io~csi/pvc-75aa0237-ef71-456b-a04c-cd8485bcf472/mount: permission denied
  Normal   Pulled          16s                kubelet  Successfully pulled image "quay.io/ocsci/nginx:latest" in 2.240981686s (2.240997379s including waiting)
  Normal   Pulling         2s (x8 over 100s)  kubelet  Pulling image "quay.io/ocsci/nginx:latest"


In the worker node on which pod is scheduled :
sh-5.1# ls -lrt /var/lib/kubelet/pods/1dc77717-b3a0-424c-a249-6a4ba29052e8/volumes/kubernetes.io~csi/
total 0
drwxr-x---. 3 root root 40 Apr 12 07:29 pvc-75aa0237-ef71-456b-a04c-cd8485bcf472
sh-5.1# 
sh-5.1# ls -lrt /var/lib/kubelet/pods/1dc77717-b3a0-424c-a249-6a4ba29052e8/volumes/kubernetes.io~csi/pvc-75aa0237-ef71-456b-a04c-cd8485bcf472/
ls: cannot access '/var/lib/kubelet/pods/1dc77717-b3a0-424c-a249-6a4ba29052e8/volumes/kubernetes.io~csi/pvc-75aa0237-ef71-456b-a04c-cd8485bcf472/mount': Permission denied
total 4
d?????????? ? ?    ?      ?            ? mount
-rw-r--r--. 1 root root 378 Apr 12 07:29 vol_data.json

Comment 3 Venky Shankar 2023-04-18 05:12:04 UTC
Milind, PTAL. I think there was another BZ similar to this related to the mount path being a symbolic link.

I'll try to dig that up. Feel free to include Madhu from CSI team for details.

Comment 4 Madhu Rajanna 2023-04-18 06:29:47 UTC
Hi Abhishek, Do we have the must-gather for this BZ and what is the exact ODF version used for testing?

Comment 5 Abhishek Mundada 2023-04-19 09:52:45 UTC
Link for the must-gather file :
https://drive.google.com/file/d/1tFsEiWNvjXAxzPwtM8-GFyP3mwqm4DrI/view?usp=sharing

ODF version :
full_version=4.13.0-121


Note You need to log in before you can comment on or make changes to this bug.