Bug 2123386 - when running with multus enabled PVs do not mount
Summary: when running with multus enabled PVs do not mount
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenShift Data Foundation
Classification: Red Hat Storage
Component: rook
Version: 4.11
Hardware: x86_64
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ODF 4.12.0
Assignee: Madhu Rajanna
QA Contact: Shivam Durgbuns
URL:
Whiteboard:
Depends On:
Blocks: 2123626
TreeView+ depends on / blocked
 
Reported: 2022-09-01 14:06 UTC by Shawn Houston
Modified: 2023-08-09 17:03 UTC (History)
6 users (show)

Fixed In Version: 4.12.0-74
Doc Type: Bug Fix
Doc Text:
Cause: When ODF is deployed in a multus enabled cluster, the CSI holder pods are missing the service account name in the template, because of this CSI holder pods are not in a Running state and CephCSI was not able to mount the CephFS/RBD PVC to the application pods. Consequence: The user is not able to mount cephfs and rbd PVC to application pods in the multus enabled cluster. Fix: Add the missing service account name to the csi holder pods. Result: The application pods should be able to mount the cephfs and rbd PVC's.
Clone Of:
: 2123626 (view as bug list)
Environment:
Last Closed: 2023-02-08 14:06:28 UTC
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github rook rook pull 10875 0 None open csi: fix holder pod creation in openshift multus cluster 2022-09-02 07:55:34 UTC

Comment 3 Madhu Rajanna 2022-09-02 06:05:38 UTC
>ls
csi-addons-controller-manager-6b68c99ff6-46jxr    openshift-worker-1testmelibvirt2smh-debug
csi-cephfsplugin-6ql7p                            openshift-worker-2testmelibvirt2smh-debug
csi-cephfsplugin-b6f7w                            openshift-worker-3testmelibvirt2smh-debug
csi-cephfsplugin-kq42r                            openshift-worker-4testmelibvirt2smh-debug
csi-cephfsplugin-pqhxf                            openshift-worker-5testmelibvirt2smh-debug
csi-cephfsplugin-provisioner-789f57d47f-n9w8s     rook-ceph-crashcollector-openshift-worker-0.testme.libvirt4gctk
csi-cephfsplugin-provisioner-789f57d47f-zq69q     rook-ceph-crashcollector-openshift-worker-2.testme.libvirtpss5w
csi-cephfsplugin-rk8vb                            rook-ceph-crashcollector-openshift-worker-3.testme.libvirtrln4z
csi-cephfsplugin-tdm2t                            rook-ceph-crashcollector-openshift-worker-4.testme.libvirt4tstn
csi-rbdplugin-cfd9f                               rook-ceph-crashcollector-openshift-worker-5.testme.libvirtthmvr
csi-rbdplugin-j5vfc                               rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-85f9bf54z7tt8
csi-rbdplugin-jht6p                               rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-59fcb555nh8vl
csi-rbdplugin-pkcz8                               rook-ceph-mgr-a-c9cb6db96-jx25q
csi-rbdplugin-provisioner-6bffcdc69c-2phkk        rook-ceph-mon-a-5fcb9566dd-4zjct
csi-rbdplugin-provisioner-6bffcdc69c-mjvkg        rook-ceph-mon-b-75d4b95f8b-x5l6z
csi-rbdplugin-v4l6w                               rook-ceph-mon-c-74c5867f94-l2cph
csi-rbdplugin-vvf4g                               rook-ceph-operator-57cbb6944b-9mk9b
must-gather-2gsml-helper                          rook-ceph-osd-0-7dd9cd7b8f-95rjw
noobaa-core-0                                     rook-ceph-osd-1-5d545c9578-rcqx9
noobaa-db-pg-0                                    rook-ceph-osd-2-84585b8f87-798hw
noobaa-operator-665944588c-lpc7l                  rook-ceph-osd-prepare-54955d151890a3817915ac92bc5a02ea-x4tql
ocs-metrics-exporter-796fc9fd5c-s4sqr             rook-ceph-osd-prepare-fce4f5a5dd008bc90a82dcc3bfcb47b7-2zvzj
ocs-operator-b57fddd65-kl5zb                      rook-ceph-osd-prepare-fdfe26392dc5dcea2e67dc360ebf8a15-pbh45
odf-console-66f9bcb487-grqt6                      rook-ceph-rgw-ocs-storagecluster-cephobjectstore-a-856c5fblwgr2
odf-operator-controller-manager-85df889d79-tvqw2  rook-ceph-tools-6765fc6564-j79pb



- apiVersion: apps/v1
  kind: DaemonSet
  metadata:
    annotations:
      deprecated.daemonset.template.generation: "1"
    creationTimestamp: "2022-09-01T12:46:40Z"
    generation: 1
    labels:
      rook-version: v4.11.0-0.cf9b14df972044635a892b2a6b498961a0d78c71
    managedFields:
    - apiVersion: apps/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:deprecated.daemonset.template.generation: {}
          f:labels:
            .: {}
            f:rook-version: {}
        f:spec:
          f:revisionHistoryLimit: {}
          f:selector: {}
          f:template:
            f:metadata:
              f:annotations:
                .: {}
                f:k8s.v1.cni.cncf.io/networks: {}
              f:labels:
                .: {}
                f:app: {}
              f:name: {}
            f:spec:
              f:affinity:
                .: {}
                f:nodeAffinity: {}
              f:containers:
                k:{"name":"csi-cephfsplugin-holder-ocs-storagecluster-cephcluster"}:
                  .: {}
                  f:args: {}
                  f:command: {}
                  f:env:
                    .: {}
                    k:{"name":"CEPH_CLUSTER_NAMESPACE"}:
                      .: {}
                      f:name: {}
                      f:value: {}
                    k:{"name":"ROOK_CEPH_CSI_DRIVER_NAME"}:
                      .: {}
                      f:name: {}
                      f:value: {}
                  f:image: {}
                  f:imagePullPolicy: {}
                  f:name: {}
                  f:resources: {}
                  f:securityContext:
                    .: {}
                    f:privileged: {}
                  f:terminationMessagePath: {}
                  f:terminationMessagePolicy: {}
                  f:volumeMounts:
                    .: {}
                    k:{"mountPath":"/var/lib/kubelet/plugins"}:
                      .: {}
                      f:mountPath: {}
                      f:mountPropagation: {}
                      f:name: {}
              f:dnsPolicy: {}
              f:hostPID: {}
              f:restartPolicy: {}
              f:schedulerName: {}
              f:securityContext: {}
              f:terminationGracePeriodSeconds: {}
              f:tolerations: {}
              f:volumes:
                .: {}
                k:{"name":"plugin-mount-dir"}:
                  .: {}
                  f:hostPath:
                    .: {}
                    f:path: {}
                    f:type: {}
                  f:name: {}
          f:updateStrategy:
            f:type: {}
      manager: Go-http-client
      operation: Update
      time: "2022-09-01T12:46:40Z"
    name: csi-cephfsplugin-holder-ocs-storagecluster-cephcluster
    namespace: openshift-storage
    resourceVersion: "43613"
    uid: 3723fa6c-f834-4c73-9e65-3ed26df53701
  spec:
    revisionHistoryLimit: 10
    selector:
      matchLabels:
        app: csi-cephfsplugin-holder
    template:
      metadata:
        annotations:
          k8s.v1.cni.cncf.io/networks: openshift-storage/odf-public-cluster
        creationTimestamp: null
        labels:
          app: csi-cephfsplugin-holder
        name: -ocs-storagecluster-cephcluster
      spec:
        affinity:
          nodeAffinity: {}
        containers:
        - args:
          - ln --force --symbolic --verbose /proc/$$$/ns/net /var/lib/kubelet/plugins/$ROOK_CEPH_CSI_DRIVER_NAME/$CEPH_CLUSTER_NAMESPACE.net.ns
            && sleep infinity
          command:
          - /bin/sh
          - -c
          env:
          - name: CEPH_CLUSTER_NAMESPACE
            value: openshift-storage
          - name: ROOK_CEPH_CSI_DRIVER_NAME
            value: openshift-storage.cephfs.csi.ceph.com
          image: registry.redhat.io/odf4/cephcsi-rhel8@sha256:7d015719d62d9a26b5a8f1ddc7b1789501b8889638b711ac3cef07e355f96a59
          imagePullPolicy: IfNotPresent
          name: csi-cephfsplugin-holder-ocs-storagecluster-cephcluster
          resources: {}
          securityContext:
            privileged: true
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
          - mountPath: /var/lib/kubelet/plugins
            mountPropagation: Bidirectional
            name: plugin-mount-dir
        dnsPolicy: ClusterFirst
        hostPID: true
        restartPolicy: Always
        schedulerName: default-scheduler
        securityContext: {}
        terminationGracePeriodSeconds: 30
        tolerations:
        - effect: NoSchedule
          key: node.ocs.openshift.io/storage
          operator: Equal
          value: "true"
        volumes:
        - hostPath:
            path: /var/lib/kubelet/plugins
            type: Directory
          name: plugin-mount-dir
    updateStrategy:
      type: OnDelete
  status:
    currentNumberScheduled: 0
    desiredNumberScheduled: 0
    numberMisscheduled: 0
    numberReady: 0




I dont see a csi holder pod created, which is responsible for creating a symlink `/proc/$$$/ns/net` to `/var/lib/kubelet/plugins/openshift-storage.rbd.csi.ceph.com/openshift-storage.net.ns` 

We don't have OCP logs to check why pods are not created for the daemonset, Need to reproduce it. Moving to Rook as no issue with cephcsi.


Note You need to log in before you can comment on or make changes to this bug.