+++ This bug was initially created as a clone of Bug #2276135 +++ Description of problem ====================== The ocs-operator should not add `reclaimspace.csiaddons.openshift.io/schedule` annotation to all openshift-* namespaces, because it could get into conflict with another operator which doesn't expect this annotation to be there. One such conflict arises when ODF is installed on ARO cluster: ARO operator is trying to get rid of this annotation on ARO namespaces (openshift-azure-loggin or openshift-azure), so that it effectively prevents ODF to be installed on ARO. The change was introduced via https://github.com/red-hat-storage/ocs-operator/pull/2074 The annotation itself seems to control ODF "reclaim space operation"[1], so there is no reason to label all openshift namespaces in the first place. [1] https://access.redhat.com/documentation/en-us/red_hat_openshift_data_foundation/4.10/html/managing_and_allocating_storage_resources/reclaiming-space-on-target-volumes_rhodf Version-Release number of selected component ============================================ ODF 4.14 How reproducible ================ 100% Steps to Reproduce ================== 1. Install ODF on ARO 4.14 cluster Actual results ============== ARO Operator is removing `reclaimspace.csiaddons.openshift.io/schedule` annotation from ARO namespaces. Example from aro-operator-master log: ``` time=2024-04-19 16:43:10.000000 level=info msg="" Update Namespace/openshift-azure-logging: u00a0u00a0&v1.Namespace{ u00a0u00a0 TypeMeta: {}, u00a0u00a0 ObjectMeta: v1.ObjectMeta{ u00a0u00a0 ... // 9 identical fields u00a0u00a0 DeletionGracePeriodSeconds: nil, u00a0u00a0 Labels: { "kubernetes.io/metadata.name": "openshift-azure-logging", "pod-security.kubernetes.io/audit": "privileged", "pod-security.kubernetes.io/enforce": "privileged", "pod-security.kubernetes.io/warn": "privileged" }, u00a0u00a0 Annotations: map[string]string{ u00a0u00a0 ... // 2 identical entries u00a0u00a0 "openshift.io/sa.scc.supplemental-groups": "1000680000/10000", u00a0u00a0 "openshift.io/sa.scc.uid-range": "1000680000/10000", -u00a0 "reclaimspace.csiaddons.openshift.io/schedule": "@weekly", u00a0u00a0 }, u00a0u00a0 OwnerReferences: { { APIVersion: "aro.openshift.io/v1alpha1", Kind: "Cluster", Name: "cluster", UID: "1b821b4f-2dcc-4ae7-ab57-452df1ad678b", ... } }, u00a0u00a0 Finalizers: nil, u00a0u00a0 ManagedFields: { { Manager: "Go-http-client", Operation: "Update", APIVersion: "v1", Time: s"2024-04-19 08:41:38 +0000 UTC", ... }, { Manager: "cluster-policy-controller", Operation: "Update", APIVersion: "v1", Time: s"2024-04-19 08:41:38 +0000 UTC", ... }, { Manager: "ocs-operator", Operation: "Update", APIVersion: "v1", Time: s"2024-04-19 16:43:10 +0000 UTC", ... } }, u00a0u00a0 }, u00a0u00a0 Spec: { Finalizers: {"kubernetes"}}, u00a0u00a0 Status: {Phase: "Active"}, u00a0u00a0 } ``` Expected results ================ ODF doesn't annotate all openshift namespaces, but only those which needs that annotation only. --- Additional comment from Martin Bukatovic on 2024-04-19 19:17:17 UTC --- See also: - https://redhat-internal.slack.com/archives/CCV9YF9PD/p1713546395182889 --- Additional comment from RHEL Program Management on 2024-04-19 19:17:31 UTC --- This bug having no release flag set previously, is now set with release flag 'odf‑4.16.0' to '?', and so is being proposed to be fixed at the ODF 4.16.0 release. Note that the 3 Acks (pm_ack, devel_ack, qa_ack), if any previously set while release flag was missing, have now been reset since the Acks are to be set against a release flag. --- Additional comment from RHEL Program Management on 2024-04-19 19:17:31 UTC --- The 'Target Release' is not to be set manually at the Red Hat OpenShift Data Foundation product. The 'Target Release' will be auto set appropriately, after the 3 Acks (pm,devel,qa) are set to "+" for a specific release flag and that release flag gets auto set to "+". --- Additional comment from Eran Tamir on 2024-04-21 04:26:57 UTC --- @muagarwa this is a blocker for ARO 4.14. Can you please prioritize it or provide a workaround we can include in ARO instructions? --- Additional comment from Madhu Rajanna on 2024-04-22 05:38:47 UTC --- The requirement was to support reclaimspace for all the namespaces starting with openshift-* https://issues.redhat.com/browse/RHSTOR-4468, @Eran i think we cannot workaround because if we add any annotation on this namespace its getting removed by the operator, if we know the exact name need to be discarded we can blocklist that in ocs-operator code. let me know if that works and what is the exact name we need to consider here. --- Additional comment from Nikhil Ladha on 2024-04-22 05:57:48 UTC --- From the slack thread it looks like "openshift-managed-upgrade-operator" and "openshift-azure-logging" are the 2 namespaces that is managed by that operator. Though, please let me know if we should be just blocklisting these 2 namespaces or how it should be? --- Additional comment from Martin Bukatovic on 2024-04-22 07:39:11 UTC --- ARO namespaces are: - openshift-azure-logging - openshift-azure-operator That said I need to point out that the design choice selected in RHSTOR-4468 is the root cause here. The proper solution is to add the annotation *only* to namespaces which are using ODF backed rbd storage. --- Additional comment from Madhu Rajanna on 2024-04-22 07:55:05 UTC --- >The proper solution is to add the annotation *only* to namespaces which are using ODF backed rbd storage. This will be a heavy thing to do for this simple use case, Fow not we are not going to hardcode anything Suggested changes in ocs-operator as below * DO restart only once when ocs-operator is starting only if the annotation is added * Skip Reconcile update operation on the namespace as it's not required. --- Additional comment from Eran Tamir on 2024-05-01 09:15:51 UTC --- Discussed offline and got to agreement on the required change. --------------------------------------------------------------------------- Cloning above BZ as a document Bug as we need to document the steps to auto reclaimspace for the RBD PVC and namespace level. The steps will be as below * Annotate the Namespace * If there are existing PVC in the namespace, restart the csi-addon operator pod @Petr/Elad Was the feature testing for manually annotating namespaces? Is it okay to test this feature now as we need to add this to documentation? @Eran/Mudit can this be a KCS until we find a new way to reclaim all the RBD PVC's by default and this can be applicable until then only?