Description of problem (please be detailed as possible and provide log snippests): When changing the ocs-operator-config configmap, modified values are not propagated automatically to the rook-ceph-operator. For example, in testing Multus, Oded and I found that changing `CSI_DISABLE_HOLDER_PODS` from "false" to "true" in the ocs-operator-config still resulted in the rook-ceph-operator running with the old value. We had to restart the rook-ceph-operator to get the configmap change to take. The behavior that causes this bug is because ocs-operator configures the rook-ceph-operator deployment with downward API references to ocs-operator-config configmap values. When a configmap value changes, the deployment spec doesn't change, and therefore the rook-ceph-operator isn't automatically restarted. See also: https://stackoverflow.com/questions/56377726/live-update-of-kubernetes-container-environment-variable-derived-from-configmapr These are all values that are likely affected by this bug: ROOK_CURRENT_NAMESPACE_ONLY CSI_DISABLE_HOLDER_PODS CSI_CLUSTER_NAME CSI_ENABLE_TOPOLOGY CSI_TOPOLOGY_DOMAIN_LABELS ROOK_CSI_ENABLE_NFS ROOK_CSI_DISABLE_DRIVER Version of all relevant components (if applicable): 4.16 Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? No, but it is a friction point for users that Eran would like to see fixed. Is there any workaround available to the best of your knowledge? Yes. Users can manually restart the rook-ceph-operator. Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 2 Can this issue reproducible? Yes. Can this issue reproduce from the UI? Yes. Steps to Reproduce: 1. install ODF 2. observe a config env var on the rook-ceph-operator pod (such as CSI_DISABLE_HOLDER_PODS) 3. modify the value observed config value in ocs-operator-config cm 4. observe the config env var on the rook-ceph-operator pod again Actual results: The config values remain the same. Expected results: Config values should change. Additional info: This is what the rook-ceph-operator pod's 'describe' output looks like after modifying CSI_DISABLE_HOLDER_PODS. ❯ oc describe pod rook-ceph-operator-84b8498cd-6qxcr Name: rook-ceph-operator-84b8498cd-6qxcr Namespace: openshift-storage Priority: 0 Service Account: rook-ceph-system Node: argo005.ceph.redhat.com/10.8.128.205 Start Time: Tue, 30 Apr 2024 09:09:04 -0600 Labels: app=rook-ceph-operator pod-template-hash=84b8498cd Annotations: alm-examples: [ { "apiVersion": "ceph.rook.io/v1", "kind": "CephCluster", "metadata": { "name": "my-rook-ceph", "namespace": "my-rook-ceph" }, "spec": { "cephVersion": { "image": "registry.redhat.io/rhceph/rhceph-7-rhel9@sha256:cda4d8682b12f13ce90211cad773100c32584b6bcea33a6cb69a66d9aece86f5" }, "dataDirHostPath": "/var/lib/rook", "mon": { "count": 3 }, "dashboard": { "enabled": true }, "network": { "hostNetwork": false }, "rbdMirroring": { "workers": 0 }, "storage": { "useAllNodes": true, "useAllDevices": true } } }, { "apiVersion": "ceph.rook.io/v1", "kind": "CephBlockPool", "metadata": { "name": "replicapool", "namespace": "my-rook-ceph" }, "spec": { "failureDomain": "host", "replicated": { "size": 3 }, "annotations": null } }, { "apiVersion": "ceph.rook.io/v1", "kind": "CephObjectStore", "metadata": { "name": "my-store", "namespace": "my-rook-ceph" }, "spec": { "metadataPool": { "failureDomain": "host", "replicated": { "size": 3 } }, "dataPool": { "failureDomain": "host", "replicated": { "size": 3 } }, "gateway": { "type": "s3", "sslCertificateRef": null, "port": 8080, "securePort": null, "instances": 1, "placement": null, "annotations": null, "resources": null } } }, { "apiVersion": "ceph.rook.io/v1", "kind": "CephObjectStoreUser", "metadata": { "name": "my-user", "namespace": "my-rook-ceph" }, "spec": { "store": "my-store", "displayName": "my display name" } }, { "apiVersion": "ceph.rook.io/v1", "kind": "CephNFS", "metadata": { "name": "my-nfs", "namespace": "rook-ceph" }, "spec": { "rados": { "pool": "myfs-data0", "namespace": "nfs-ns" }, "server": { "active": 3, "placement": null, "annotations": null, "resources": null } } }, { "apiVersion": "ceph.rook.io/v1", "kind": "CephClient", "metadata": { "name": "cinder", "namespace": "rook-ceph" }, "spec": { "caps": { "mon": "profile rbd", "osd": "profile rbd pool=volumes, profile rbd pool=vms, profile rbd-read-only pool=images" } } }, { "apiVersion": "ceph.rook.io/v1", "kind": "CephFilesystem", "metadata": { "name": "myfs", "namespace": "rook-ceph" }, "spec": { "dataPools": [ { "compressionMode": "", "crushRoot": "", "deviceClass": "", "erasureCoded": { "algorithm": "", "codingChunks": 0, "dataChunks": 0 }, "failureDomain": "host", "replicated": { "requireSafeReplicaSize": false, "size": 1, "targetSizeRatio": 0.5 } } ], "metadataPool": { "compressionMode": "", "crushRoot": "", "deviceClass": "", "erasureCoded": { "algorithm": "", "codingChunks": 0, "dataChunks": 0 }, "failureDomain": "", "replicated": { "requireSafeReplicaSize": false, "size": 1, "targetSizeRatio": 0 } }, "metadataServer": { "activeCount": 1, "activeStandby": true, "placement": {}, "resources": {} }, "preservePoolsOnDelete": false, "preserveFilesystemOnDelete": false } }, { "apiVersion": "ceph.rook.io/v1", "kind": "CephRBDMirror", "metadata": { "name": "my-rbd-mirror", "namespace": "rook-ceph" }, "spec": { "annotations": null, "count": 1, "placement": { "topologyKey": "kubernetes.io/hostname" }, "resources": null } } ] capabilities: Basic Install containerImage: registry.redhat.io/odf4/rook-ceph-rhel9-operator@sha256:512ee873eb3d7524814e06f23ba88de926ce5c8a16cccf5a487e0efa8686741f externalClusterScript: '' # removed for brevity features.operators.openshift.io/disconnected: true features.operators.openshift.io/fips-compliant: true features.operators.openshift.io/proxy-aware: true features.operators.openshift.io/tls-profiles: false features.operators.openshift.io/token-auth-aws: false features.operators.openshift.io/token-auth-azure: false features.operators.openshift.io/token-auth-gcp: false k8s.ovn.org/pod-networks: {"default":{"ip_addresses":["10.128.2.70/23"],"mac_address":"0a:58:0a:80:02:46","gateway_ips":["10.128.2.1"],"routes":[{"dest":"10.128.0.0... k8s.v1.cni.cncf.io/network-status: [{ "name": "ovn-kubernetes", "interface": "eth0", "ips": [ "10.128.2.70" ], "mac": "0a:58:0a:80:02:46", "default": true, "dns": {} }] olm.operatorGroup: openshift-storage-nn2fh olm.operatorNamespace: openshift-storage olm.skipRange: >=4.2.0 <4.16.0-90.stable olm.targetNamespaces: openshift-storage openshift.io/scc: rook-ceph operatorframework.io/properties: {"properties":[{"type":"olm.gvk","value":{"group":"ceph.rook.io","kind":"CephBlockPool","version":"v1"}},{"type":"olm.gvk","value":{"group... operators.openshift.io/valid-subscription: ["OpenShift Platform Plus","OpenShift Data Foundation Essentials","OpenShift Data Foundation Advanced"] operators.operatorframework.io/builder: operator-sdk-v1.25.0 operators.operatorframework.io/operator-type: non-standalone operators.operatorframework.io/project_layout: unknown repository: https://github.com/red-hat-storage/rook tectonic-visibility: ocs Status: Running IP: 10.128.2.70 IPs: IP: 10.128.2.70 Controlled By: ReplicaSet/rook-ceph-operator-84b8498cd Containers: rook-ceph-operator: Container ID: cri-o://3873c7238ef7af71b2054a9c3840e273acfcccb7bf6da212ad39e46712853b13 Image: registry.redhat.io/odf4/rook-ceph-rhel9-operator@sha256:512ee873eb3d7524814e06f23ba88de926ce5c8a16cccf5a487e0efa8686741f Image ID: registry.redhat.io/odf4/rook-ceph-rhel9-operator@sha256:03d9d8fbe6e202aa5a756e28333c11fbb98d113adb58cc8a21196aee81ed179b Port: <none> Host Port: <none> Args: ceph operator State: Running Started: Tue, 30 Apr 2024 09:09:26 -0600 Ready: True Restart Count: 0 Environment: ROOK_CURRENT_NAMESPACE_ONLY: <set to the key 'ROOK_CURRENT_NAMESPACE_ONLY' of config map 'ocs-operator-config'> Optional: false CSI_DISABLE_HOLDER_PODS: <set to the key 'CSI_DISABLE_HOLDER_PODS' of config map 'ocs-operator-config'> Optional: false ROOK_ALLOW_MULTIPLE_FILESYSTEMS: false ROOK_LOG_LEVEL: INFO ROOK_CEPH_STATUS_CHECK_INTERVAL: 60s ROOK_MON_HEALTHCHECK_INTERVAL: 45s ROOK_MON_OUT_TIMEOUT: 600s ROOK_DISCOVER_DEVICES_INTERVAL: 60m ROOK_HOSTPATH_REQUIRES_PRIVILEGED: true ROOK_ENABLE_SELINUX_RELABELING: true ROOK_ENABLE_FSGROUP: true ROOK_ENABLE_FLEX_DRIVER: false ROOK_ENABLE_DISCOVERY_DAEMON: false ROOK_ENABLE_MACHINE_DISRUPTION_BUDGET: false ROOK_DISABLE_DEVICE_HOTPLUG: true ROOK_CSI_ALLOW_UNSUPPORTED_VERSION: true ROOK_DISABLE_ADMISSION_CONTROLLER: true ROOK_CSIADDONS_IMAGE: registry.redhat.io/odf4/odf-csi-addons-sidecar-rhel9@sha256:d0ca282694892d6caf025a35a593a3633785d2a40f4f8984e7f94a6906bb4236 ROOK_CSI_CEPH_IMAGE: registry.redhat.io/odf4/cephcsi-rhel9@sha256:d851bc4896e3666ba4d965eac89010ed5eea6c59d55027a5f5a01f9b079aeafe ROOK_CSI_REGISTRAR_IMAGE: registry.redhat.io/openshift4/ose-csi-node-driver-registrar@sha256:b7eacc160fcce0881a00be2eb8d050a66b6cf68bcac2ef9da72d7c0297f77c0f ROOK_CSI_RESIZER_IMAGE: registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:927629fd0731988d52d5bb1094b650bc5def609bacb406dac5e60905e4c9ca26 ROOK_CSI_PROVISIONER_IMAGE: registry.redhat.io/openshift4/ose-csi-external-provisioner@sha256:2da32b524163a1e046bdde7750fe71a2f1175e509357db3cd1300ef849f4f0b6 ROOK_CSI_SNAPSHOTTER_IMAGE: registry.redhat.io/openshift4/ose-csi-external-snapshotter@sha256:965111171af569965e07b724eb93ea77077c6272023c02d0f1aa80ebcdef48fa ROOK_CSI_ATTACHER_IMAGE: registry.redhat.io/openshift4/ose-csi-external-attacher@sha256:bce20ed64dbee694666b75a96fd505223e8eed193d5cd40a607d871d0cc8b9c0 ROOK_OBC_PROVISIONER_NAME_PREFIX: openshift-storage CSI_ENABLE_METADATA: false CSI_PLUGIN_PRIORITY_CLASSNAME: system-node-critical CSI_PROVISIONER_PRIORITY_CLASSNAME: system-cluster-critical CSI_CLUSTER_NAME: <set to the key 'CSI_CLUSTER_NAME' of config map 'ocs-operator-config'> Optional: false CSI_DRIVER_NAME_PREFIX: openshift-storage CSI_ENABLE_TOPOLOGY: <set to the key 'CSI_ENABLE_TOPOLOGY' of config map 'ocs-operator-config'> Optional: false CSI_TOPOLOGY_DOMAIN_LABELS: <set to the key 'CSI_TOPOLOGY_DOMAIN_LABELS' of config map 'ocs-operator-config'> Optional: false ROOK_CSI_ENABLE_NFS: <set to the key 'ROOK_CSI_ENABLE_NFS' of config map 'ocs-operator-config'> Optional: false ROOK_CSI_DISABLE_DRIVER: <set to the key 'ROOK_CSI_DISABLE_DRIVER' of config map 'ocs-operator-config'> Optional: false CSI_PROVISIONER_TOLERATIONS: - key: node.ocs.openshift.io/storage operator: Equal value: "true" effect: NoSchedule CSI_PLUGIN_TOLERATIONS: - key: node.ocs.openshift.io/storage operator: Equal value: "true" effect: NoSchedule CSI_LOG_LEVEL: 5 CSI_SIDECAR_LOG_LEVEL: 1 CSI_ENABLE_CSIADDONS: true NODE_NAME: (v1:spec.nodeName) POD_NAME: rook-ceph-operator-84b8498cd-6qxcr (v1:metadata.name) POD_NAMESPACE: openshift-storage (v1:metadata.namespace) ROOK_OBC_WATCH_OPERATOR_NAMESPACE: true OPERATOR_CONDITION_NAME: rook-ceph-operator.v4.16.0-90.stable Mounts: /etc/ceph from default-config-dir (rw) /var/lib/rook from rook-config (rw) /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-stz5g (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: rook-config: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> default-config-dir: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: SizeLimit: <unset> kube-api-access-stz5g: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: <nil> DownwardAPI: true ConfigMapName: openshift-service-ca.crt ConfigMapOptional: <nil> QoS Class: BestEffort Node-Selectors: <none> Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s node.ocs.openshift.io/storage=true:NoSchedule Events: <none>
Full procedure: https://docs.google.com/document/d/1CFvmSun2rbIpol0rmNht1AfXkEz7WouqB_XSiVrlT0c/edit