Created attachment 1804383 [details]
ocs-operator logs

By default the globalmount path is /var/lib/kubelet this is failing because here the globalmount path is changed to /var/data/kubelet

Do we know why the mount path is changed?

Please follow 
[1] https://cloud.ibm.com/docs/openshift?topic=openshift-ocs-storage-install

We're using the azure CSI Driver to provision remote block volumes. The mon PVCs and PVs were successfully created and bound.

Hi, in IBM ROKS, for OCS, we create a ConfigMap to change the Kubelet path for OCS to /var/data/kubelet. Like this

% oc describe cm rook-ceph-operator-config -n openshift-storage
Name:         rook-ceph-operator-config
Namespace:    openshift-storage
Labels:       <none>
Annotations:  <none>

Data
====
ROOK_CSI_KUBELET_DIR_PATH:
----
/var/data/kubelet
Events:  <none>

Ok, if you are using ROOK_CSI_KUBELET_DIR_PATH then the problem lies somewhere else.
Because its mon pods, its better if rook guys take a look.

Travis, could someone look at this?

Shrisha Can we get a must-gather for more info? For example, we at least need:
- CephCluster CR
- Mon PVCs

But it sounds like something isn't working with the azure volumes being mounted in the mons. Are you able to create a test pod with a pvc from the same azure storage class?

Travis,
Yes I was able to create a test pod and mount a PVC of the same storage class successfully.
This is the cephcluster CR :

 % oc get cephcluster -n openshift-storage -o yaml
apiVersion: v1
items:
- apiVersion: ceph.rook.io/v1
  kind: CephCluster
  metadata:
    creationTimestamp: "2021-07-22T07:31:07Z"
    finalizers:
    - cephcluster.ceph.rook.io
    generation: 1
    labels:
      app: ocs-storagecluster
    managedFields:
    - apiVersion: ceph.rook.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:labels:
            .: {}
            f:app: {}
          f:ownerReferences:
            .: {}
            k:{"uid":"3ca890d2-ddb7-4209-aaa3-6f5c779d77ad"}:
              .: {}
              f:apiVersion: {}
              f:blockOwnerDeletion: {}
              f:controller: {}
              f:kind: {}
              f:name: {}
              f:uid: {}
        f:spec:
          .: {}
          f:cephVersion:
            .: {}
            f:image: {}
          f:cleanupPolicy:
            .: {}
            f:sanitizeDisks: {}
          f:continueUpgradeAfterChecksEvenIfNotHealthy: {}
          f:crashCollector:
            .: {}
            f:disable: {}
          f:dashboard: {}
          f:dataDirHostPath: {}
          f:disruptionManagement:
            .: {}
            f:machineDisruptionBudgetNamespace: {}
            f:managePodBudgets: {}
          f:external:
            .: {}
            f:enable: {}
          f:healthCheck:
            .: {}
            f:daemonHealth: {}
          f:logCollector:
            .: {}
            f:enabled: {}
            f:periodicity: {}
          f:mgr:
            .: {}
            f:modules: {}
          f:mon:
            .: {}
            f:count: {}
            f:volumeClaimTemplate:
              .: {}
              f:metadata: {}
              f:spec: {}
              f:status: {}
          f:monitoring:
            .: {}
            f:enabled: {}
            f:rulesNamespace: {}
          f:network:
            .: {}
            f:hostNetwork: {}
            f:provider: {}
            f:selectors: {}
          f:placement:
            .: {}
            f:all: {}
            f:arbiter: {}
            f:mon: {}
          f:removeOSDsIfOutAndSafeToRemove: {}
          f:resources:
            .: {}
            f:mds: {}
            f:mgr: {}
            f:mon: {}
            f:rgw: {}
          f:security:
            .: {}
            f:kms: {}
          f:storage:
            .: {}
            f:config: {}
            f:storageClassDeviceSets: {}
      manager: ocs-operator
      operation: Update
      time: "2021-07-22T07:31:07Z"
    - apiVersion: ceph.rook.io/v1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:finalizers:
            .: {}
            v:"cephcluster.ceph.rook.io": {}
        f:status:
          .: {}
          f:conditions: {}
          f:message: {}
          f:phase: {}
          f:state: {}
          f:version: {}
      manager: rook
      operation: Update
      time: "2021-07-22T07:31:13Z"
    name: ocs-storagecluster-cephcluster
    namespace: openshift-storage
    ownerReferences:
    - apiVersion: ocs.openshift.io/v1
      blockOwnerDeletion: true
      controller: true
      kind: StorageCluster
      name: ocs-storagecluster
      uid: 3ca890d2-ddb7-4209-aaa3-6f5c779d77ad
    resourceVersion: "2837895"
    selfLink: /apis/ceph.rook.io/v1/namespaces/openshift-storage/cephclusters/ocs-storagecluster-cephcluster
    uid: ebd50e6c-97af-42a6-b6b6-eee525d2a2de
  spec:
    cephVersion:
      image: registry.redhat.io/rhceph/rhceph-4-rhel8@sha256:725f93133acc0fb1ca845bd12e77f20d8629cad0e22d46457b2736578698eb6c
    cleanupPolicy:
      sanitizeDisks: {}
    continueUpgradeAfterChecksEvenIfNotHealthy: true
    crashCollector:
      disable: false
    dashboard: {}
    dataDirHostPath: /var/lib/rook
    disruptionManagement:
      machineDisruptionBudgetNamespace: openshift-machine-api
      managePodBudgets: true
    external:
      enable: false
    healthCheck:
      daemonHealth:
        mon:
          timeout: 15m
        osd: {}
        status: {}
    logCollector:
      enabled: true
      periodicity: 24h
    mgr:
      modules:
      - enabled: true
        name: pg_autoscaler
      - enabled: true
        name: balancer
    mon:
      count: 3
      volumeClaimTemplate:
        metadata:
          creationTimestamp: null
        spec:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 20Gi
          storageClassName: sat-azure-block-gold-metro
          volumeMode: Filesystem
        status: {}
    monitoring:
      enabled: true
      rulesNamespace: openshift-storage
    network:
      hostNetwork: false
      provider: ""
      selectors: null
    placement:
      all:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: cluster.ocs.openshift.io/openshift-storage
                operator: Exists
        tolerations:
        - effect: NoSchedule
          key: node.ocs.openshift.io/storage
          operator: Equal
          value: "true"
      arbiter:
        tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/master
          operator: Exists
      mon:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: cluster.ocs.openshift.io/openshift-storage
                operator: Exists
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - rook-ceph-mon
            topologyKey: topology.rook.io/rack
    removeOSDsIfOutAndSafeToRemove: false
    resources:
      mds:
        limits:
          cpu: "3"
          memory: 8Gi
        requests:
          cpu: "3"
          memory: 8Gi
      mgr:
        limits:
          cpu: "1"
          memory: 3Gi
        requests:
          cpu: "1"
          memory: 3Gi
      mon:
        limits:
          cpu: "1"
          memory: 2Gi
        requests:
          cpu: "1"
          memory: 2Gi
      rgw:
        limits:
          cpu: "2"
          memory: 4Gi
        requests:
          cpu: "2"
          memory: 4Gi
    security:
      kms: {}
    storage:
      config: null
      storageClassDeviceSets:
      - count: 1
        name: ocs-deviceset-0
        placement:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: cluster.ocs.openshift.io/openshift-storage
                  operator: Exists
          tolerations:
          - effect: NoSchedule
            key: node.ocs.openshift.io/storage
            operator: Equal
            value: "true"
          topologySpreadConstraints:
          - labelSelector:
              matchExpressions:
              - key: ceph.rook.io/pvc
                operator: Exists
            maxSkew: 1
            topologyKey: topology.rook.io/rack
            whenUnsatisfiable: DoNotSchedule
          - labelSelector:
              matchExpressions:
              - key: ceph.rook.io/pvc
                operator: Exists
            maxSkew: 1
            topologyKey: kubernetes.io/hostname
            whenUnsatisfiable: ScheduleAnyway
        portable: true
        preparePlacement:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: cluster.ocs.openshift.io/openshift-storage
                  operator: Exists
          tolerations:
          - effect: NoSchedule
            key: node.ocs.openshift.io/storage
            operator: Equal
            value: "true"
          topologySpreadConstraints:
          - labelSelector:
              matchExpressions:
              - key: ceph.rook.io/pvc
                operator: Exists
            maxSkew: 1
            topologyKey: topology.rook.io/rack
            whenUnsatisfiable: DoNotSchedule
          - labelSelector:
              matchExpressions:
              - key: ceph.rook.io/pvc
                operator: Exists
            maxSkew: 1
            topologyKey: kubernetes.io/hostname
            whenUnsatisfiable: ScheduleAnyway
        resources:
          limits:
            cpu: "2"
            memory: 5Gi
          requests:
            cpu: "2"
            memory: 5Gi
        tuneFastDeviceClass: true
        volumeClaimTemplates:
        - metadata:
            annotations:
              crushDeviceClass: ""
            creationTimestamp: null
          spec:
            accessModes:
            - ReadWriteOnce
            resources:
              requests:
                storage: 100Gi
            storageClassName: sat-azure-block-gold-metro
            volumeMode: Block
          status: {}
      - count: 1
        name: ocs-deviceset-1
        placement:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: cluster.ocs.openshift.io/openshift-storage
                  operator: Exists
          tolerations:
          - effect: NoSchedule
            key: node.ocs.openshift.io/storage
            operator: Equal
            value: "true"
          topologySpreadConstraints:
          - labelSelector:
              matchExpressions:
              - key: ceph.rook.io/pvc
                operator: Exists
            maxSkew: 1
            topologyKey: topology.rook.io/rack
            whenUnsatisfiable: DoNotSchedule
          - labelSelector:
              matchExpressions:
              - key: ceph.rook.io/pvc
                operator: Exists
            maxSkew: 1
            topologyKey: kubernetes.io/hostname
            whenUnsatisfiable: ScheduleAnyway
        portable: true
        preparePlacement:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: cluster.ocs.openshift.io/openshift-storage
                  operator: Exists
          tolerations:
          - effect: NoSchedule
            key: node.ocs.openshift.io/storage
            operator: Equal
            value: "true"
          topologySpreadConstraints:
          - labelSelector:
              matchExpressions:
              - key: ceph.rook.io/pvc
                operator: Exists
            maxSkew: 1
            topologyKey: topology.rook.io/rack
            whenUnsatisfiable: DoNotSchedule
          - labelSelector:
              matchExpressions:
              - key: ceph.rook.io/pvc
                operator: Exists
            maxSkew: 1
            topologyKey: kubernetes.io/hostname
            whenUnsatisfiable: ScheduleAnyway
        resources:
          limits:
            cpu: "2"
            memory: 5Gi
          requests:
            cpu: "2"
            memory: 5Gi
        tuneFastDeviceClass: true
        volumeClaimTemplates:
        - metadata:
            annotations:
              crushDeviceClass: ""
            creationTimestamp: null
          spec:
            accessModes:
            - ReadWriteOnce
            resources:
              requests:
                storage: 100Gi
            storageClassName: sat-azure-block-gold-metro
            volumeMode: Block
          status: {}
      - count: 1
        name: ocs-deviceset-2
        placement:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: cluster.ocs.openshift.io/openshift-storage
                  operator: Exists
          tolerations:
          - effect: NoSchedule
            key: node.ocs.openshift.io/storage
            operator: Equal
            value: "true"
          topologySpreadConstraints:
          - labelSelector:
              matchExpressions:
              - key: ceph.rook.io/pvc
                operator: Exists
            maxSkew: 1
            topologyKey: topology.rook.io/rack
            whenUnsatisfiable: DoNotSchedule
          - labelSelector:
              matchExpressions:
              - key: ceph.rook.io/pvc
                operator: Exists
            maxSkew: 1
            topologyKey: kubernetes.io/hostname
            whenUnsatisfiable: ScheduleAnyway
        portable: true
        preparePlacement:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
              - matchExpressions:
                - key: cluster.ocs.openshift.io/openshift-storage
                  operator: Exists
          tolerations:
          - effect: NoSchedule
            key: node.ocs.openshift.io/storage
            operator: Equal
            value: "true"
          topologySpreadConstraints:
          - labelSelector:
              matchExpressions:
              - key: ceph.rook.io/pvc
                operator: Exists
            maxSkew: 1
            topologyKey: topology.rook.io/rack
            whenUnsatisfiable: DoNotSchedule
          - labelSelector:
              matchExpressions:
              - key: ceph.rook.io/pvc
                operator: Exists
            maxSkew: 1
            topologyKey: kubernetes.io/hostname
            whenUnsatisfiable: ScheduleAnyway
        resources:
          limits:
            cpu: "2"
            memory: 5Gi
          requests:
            cpu: "2"
            memory: 5Gi
        tuneFastDeviceClass: true
        volumeClaimTemplates:
        - metadata:
            annotations:
              crushDeviceClass: ""
            creationTimestamp: null
          spec:
            accessModes:
            - ReadWriteOnce
            resources:
              requests:
                storage: 100Gi
            storageClassName: sat-azure-block-gold-metro
            volumeMode: Block
          status: {}
  status:
    conditions:
    - lastHeartbeatTime: "2021-07-22T07:31:13Z"
      lastTransitionTime: "2021-07-22T07:31:13Z"
      message: Cluster is creating
      reason: ClusterProgressing
      status: "True"
      type: Progressing
    - lastHeartbeatTime: "2021-07-22T07:41:25Z"
      lastTransitionTime: "2021-07-22T07:41:25Z"
      message: Failed to create cluster
      reason: ClusterFailure
      status: "True"
      type: Failure
    message: Failed to create cluster
    phase: Failure
    state: Error
    version:
      image: registry.redhat.io/rhceph/rhceph-4-rhel8@sha256:725f93133acc0fb1ca845bd12e77f20d8629cad0e22d46457b2736578698eb6c
      version: 14.2.11-181
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

And this is the PVC :

% oc get pvc rook-ceph-mon-a -n openshift-storage -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: disk.csi.azure.com
    volume.kubernetes.io/selected-node: azure-vms000004
  creationTimestamp: "2021-07-22T07:31:18Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: rook-ceph-mon
    ceph-version: 14.2.11-181
    ceph_daemon_id: a
    ceph_daemon_type: mon
    mon: a
    mon_canary: "true"
    mon_cluster: openshift-storage
    pvc_name: rook-ceph-mon-a
    rook-version: 4.7-145.a418c74.release_4.7
    rook_cluster: openshift-storage
  managedFields:
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:labels:
          .: {}
          f:app: {}
          f:ceph-version: {}
          f:ceph_daemon_id: {}
          f:ceph_daemon_type: {}
          f:mon: {}
          f:mon_canary: {}
          f:mon_cluster: {}
          f:pvc_name: {}
          f:rook-version: {}
          f:rook_cluster: {}
        f:ownerReferences:
          .: {}
          k:{"uid":"ebd50e6c-97af-42a6-b6b6-eee525d2a2de"}:
            .: {}
            f:apiVersion: {}
            f:blockOwnerDeletion: {}
            f:controller: {}
            f:kind: {}
            f:name: {}
            f:uid: {}
      f:spec:
        f:accessModes: {}
        f:resources:
          f:requests:
            .: {}
            f:storage: {}
        f:storageClassName: {}
        f:volumeMode: {}
    manager: rook
    operation: Update
    time: "2021-07-22T07:31:18Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:volume.kubernetes.io/selected-node: {}
    manager: kube-scheduler
    operation: Update
    time: "2021-07-22T07:31:19Z"
  - apiVersion: v1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:pv.kubernetes.io/bind-completed: {}
          f:pv.kubernetes.io/bound-by-controller: {}
          f:volume.beta.kubernetes.io/storage-provisioner: {}
      f:spec:
        f:volumeName: {}
      f:status:
        f:accessModes: {}
        f:capacity:
          .: {}
          f:storage: {}
        f:phase: {}
    manager: kube-controller-manager
    operation: Update
    time: "2021-07-22T07:31:21Z"
  name: rook-ceph-mon-a
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: ceph.rook.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: CephCluster
    name: ocs-storagecluster-cephcluster
    uid: ebd50e6c-97af-42a6-b6b6-eee525d2a2de
  resourceVersion: "2086629"
  selfLink: /api/v1/namespaces/openshift-storage/persistentvolumeclaims/rook-ceph-mon-a
  uid: f7cb5eb2-5231-4ba2-9757-2bed3cf0207e
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: sat-azure-block-gold-metro
  volumeMode: Filesystem
  volumeName: pvc-f7cb5eb2-5231-4ba2-9757-2bed3cf0207e
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 20Gi
  phase: Bound

Please find the must-gather for the cluster here : https://drive.google.com/file/d/1Hmxk-G8J_olGqJI0nGgVkfNeGm9KNt-d/view?usp=sharing

The interesting point here is that the mon PVCs are bound to their PVs and successfully mounted in the mon canary pods, so we know the volumes are basically working. Then rook deletes the canary pods and starts up the mon daemon pods, where these volumes claim they don't exist. 

Shirisha Could you check on the following?
- What nodes are the mon-canary pods assigned to? Are the mon daemon pods starting on the same nodes or different nodes?
- If you create another test pod with a test volume, then delete the pod (without deleting the pvc), then create a new pod, does it start successfully? Or does that test pod hit the same issue after the volume is re-used? Or is it a factor if the pod starts on the same node or a different node?
- Does it change anything if you change the reclaimPolicy on the azure storage class to retain?

Not a 4.8 blocker

Hi Travis,

There was a mount propagation issue with the installed azure driver and fixing that fixed this issue.
Thank you for looking into it.