Bug 2276135

Summary: ocs-operator should not be annotating all openshift-* namespaces
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Martin Bukatovic <mbukatov>
Component: ocs-operatorAssignee: Nikhil Ladha <nladha>
Status: CLOSED ERRATA QA Contact: Yuli Persky <ypersky>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.14CC: etamir, mrajanna, muagarwa, nladha, odf-bz-bot
Target Milestone: ---   
Target Release: ODF 4.16.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: 4.16.0-92 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 2278568 2278641 2278642 (view as bug list) Environment:
Last Closed: 2024-07-17 13:20:03 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2278568, 2278641, 2278642    

Description Martin Bukatovic 2024-04-19 19:17:17 UTC
Description of problem
======================

The ocs-operator should not add `reclaimspace.csiaddons.openshift.io/schedule`
annotation to all openshift-* namespaces, because it could get into conflict
with another operator which doesn't expect this annotation to be there.

One such conflict arises when ODF is installed on ARO cluster: ARO operator is
trying to get rid of this annotation on ARO namespaces (openshift-azure-loggin
or openshift-azure), so that it effectively prevents ODF to be installed on ARO.

The change was introduced via
https://github.com/red-hat-storage/ocs-operator/pull/2074

The annotation itself seems to control ODF "reclaim space operation"[1], so
there is no reason to label all openshift namespaces in the first place.

[1] https://access.redhat.com/documentation/en-us/red_hat_openshift_data_foundation/4.10/html/managing_and_allocating_storage_resources/reclaiming-space-on-target-volumes_rhodf

Version-Release number of selected component
============================================

ODF 4.14

How reproducible
================

100%

Steps to Reproduce
==================

1. Install ODF on ARO 4.14 cluster

Actual results
==============

ARO Operator is removing `reclaimspace.csiaddons.openshift.io/schedule`
annotation from ARO namespaces.

Example from aro-operator-master log:

```
time=2024-04-19 16:43:10.000000 level=info msg=""
Update Namespace/openshift-azure-logging: u00a0u00a0&v1.Namespace{
    u00a0u00a0 TypeMeta: {},
    u00a0u00a0 ObjectMeta: v1.ObjectMeta{
        u00a0u00a0             ... // 9 identical fields
        u00a0u00a0             DeletionGracePeriodSeconds: nil,
        u00a0u00a0             Labels:                     {
            "kubernetes.io/metadata.name": "openshift-azure-logging",
            "pod-security.kubernetes.io/audit": "privileged",
            "pod-security.kubernetes.io/enforce": "privileged",
            "pod-security.kubernetes.io/warn": "privileged"
        },
        u00a0u00a0             Annotations: map[string]string{
            u00a0u00a0                 ... // 2 identical entries
            u00a0u00a0                 "openshift.io/sa.scc.supplemental-groups":	"1000680000/10000",
            u00a0u00a0                 "openshift.io/sa.scc.uid-range":                "1000680000/10000",
            -u00a0                     "reclaimspace.csiaddons.openshift.io/schedule": "@weekly",
            u00a0u00a0
        },
        u00a0u00a0             OwnerReferences: {
                {
          	 APIVersion: "aro.openshift.io/v1alpha1",
                Kind: "Cluster",
                Name: "cluster",	
                UID: "1b821b4f-2dcc-4ae7-ab57-452df1ad678b",                     	
                ...
            }
        },
        u00a0u00a0             Finalizers:	 nil,
        u00a0u00a0             ManagedFields:   {
                {
                Manager: "Go-http-client",
                Operation: "Update",
                APIVersion: "v1",
                Time: s"2024-04-19 08:41:38 +0000 UTC",
                ...
            },
          	 {
                Manager: "cluster-policy-controller",
                Operation: "Update",
                APIVersion: "v1",
                Time: s"2024-04-19 08:41:38 +0000 UTC",
                ...
            },
                {
                Manager: "ocs-operator",
                Operation: "Update",
                APIVersion: "v1",
                Time: s"2024-04-19 16:43:10 +0000 UTC",
                ...
            }
        },	
        u00a0u00a0
    },
    u00a0u00a0 Spec:   {
        Finalizers: {"kubernetes"}},
    u00a0u00a0 Status: {Phase: "Active"},
    u00a0u00a0
}       
```

Expected results
================

ODF doesn't annotate all openshift namespaces, but only those which needs that annotation only.

Comment 7 Martin Bukatovic 2024-04-22 07:39:11 UTC
ARO namespaces are:

- openshift-azure-logging
- openshift-azure-operator

That said I need to point out that the design choice selected in RHSTOR-4468 is the root cause here.

The proper solution is to add the annotation *only* to namespaces which are using ODF backed rbd storage.

Comment 13 Yuli Persky 2024-06-03 07:05:51 UTC
A clarification - This BZ is Azure specific bug. It was not seen on any of the other platform. 

For verification : the following command should be run on some ( no need to test all) of the openshift-* namespaces: 

oc get ns openshift-storage -o yaml | grep "reclaimspace"

same BZ for 4.14 : https://bugzilla.redhat.com/show_bug.cgi?id=2278642

same BZ for 4.15.3 : https://bugzilla.redhat.com/show_bug.cgi?id=2278641


In order to verify this BZ - an ARO cluster should be deployed and the verification should be performed on the ARO cluster.

Comment 14 Madhu Rajanna 2024-06-03 07:39:06 UTC
Hi Yuli,
Yes the problem was seen in ARO but for verification, we don't need ARO, we just need to ensure that we are not annotating the namespaces anymore in any clusters.

Comment 16 errata-xmlrpc 2024-07-17 13:20:03 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat OpenShift Data Foundation 4.16.0 security, enhancement & bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:4591