Bug 2319352 - [ODF-4.17.z] Cannot delete the namespace after uninstalling ocs-client-operator and other dependency operators
Summary: [ODF-4.17.z] Cannot delete the namespace after uninstalling ocs-client-operat...
Keywords:
Status: NEW
Alias: None
Product: Red Hat OpenShift Data Foundation
Classification: Red Hat Storage
Component: ocs-operator
Version: 4.17
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: ---
Assignee: Mudit Agarwal
QA Contact: Elad
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2024-10-17 10:21 UTC by Kaustav Majumder
Modified: 2024-10-17 10:22 UTC (History)
1 user (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker OCSBZM-9400 0 None None None 2024-10-17 10:22:51 UTC

Description Kaustav Majumder 2024-10-17 10:21:06 UTC
This bug was initially created as a copy of Bug #2318506

I am copying this bug because: 



Description of problem:
After uninstalling ocs-client-operator and its dependent operators, the namespace 'openshift-storage-client' cannot be deleted. This is the namespace where the operator was installed. This is tested on client cluster in a provider-client setup.

% oc get ns openshift-storage-client -o yaml
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    openshift.io/sa.scc.mcs: s0:c25,c20
    openshift.io/sa.scc.supplemental-groups: 1000640000/10000
    openshift.io/sa.scc.uid-range: 1000640000/10000
  creationTimestamp: "2024-10-14T05:04:29Z"
  deletionTimestamp: "2024-10-14T09:12:00Z"
  labels:
    kubernetes.io/metadata.name: openshift-storage-client
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/audit-version: v1.24
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: v1.24
    pod-security.kubernetes.io/warn: restricted
    pod-security.kubernetes.io/warn-version: v1.24
    security.openshift.io/scc.podSecurityLabelSync: "true"
  name: openshift-storage-client
  resourceVersion: "87170"
  uid: 059a125c-da2c-4829-a0c1-f31ba9354a8f
spec:
  finalizers:
  - kubernetes
status:
  conditions:
  - lastTransitionTime: "2024-10-14T09:12:07Z"
    message: All resources successfully discovered
    reason: ResourcesDiscovered
    status: "False"
    type: NamespaceDeletionDiscoveryFailure
  - lastTransitionTime: "2024-10-14T09:12:07Z"
    message: All legacy kube types successfully parsed
    reason: ParsedGroupVersions
    status: "False"
    type: NamespaceDeletionGroupVersionParsingFailure
  - lastTransitionTime: "2024-10-14T09:12:07Z"
    message: 'Failed to delete all resource types, 1 remaining: admission webhook
      "admissionwebhook.noobaa.io" denied the request: Noobaa cleanup policy is not
      set, blocking Noobaa deletion'
    reason: ContentDeletionFailed
    status: "True"
    type: NamespaceDeletionContentFailure
  - lastTransitionTime: "2024-10-14T09:12:07Z"
    message: All content successfully removed
    reason: ContentRemoved
    status: "False"
    type: NamespaceContentRemaining
  - lastTransitionTime: "2024-10-14T09:12:07Z"
    message: All content-preserving finalizers finished
    reason: ContentHasNoFinalizers
    status: "False"
    type: NamespaceFinalizersRemaining
  phase: Terminating


% oc get Noobaa
NAME            S3-ENDPOINTS   STS-ENDPOINTS   SYSLOG-ENDPOINTS   IMAGE   PHASE      AGE
noobaa-remote                                                             Rejected   77m

% oc get Noobaa noobaa-remote -o yaml -n openshift-storage-client
apiVersion: noobaa.io/v1alpha1
kind: NooBaa
metadata:
  annotations:
    remote-client-noobaa: "true"
  creationTimestamp: "2024-10-14T05:59:03Z"
  deletionGracePeriodSeconds: 0
  deletionTimestamp: "2024-10-14T07:12:26Z"
  finalizers:
  - noobaa.io/graceful_finalizer
  generation: 3
  labels:
    app: noobaa
  name: noobaa-remote
  namespace: openshift-storage-client
  ownerReferences:
  - apiVersion: ocs.openshift.io/v1alpha1
    kind: StorageClient
    name: storage-client
    uid: 6f40c83c-1fb9-4691-81d6-8a5ff6a6f0e6
  resourceVersion: "51471"
  uid: cf6140a5-44a8-4781-a9f4-fa70284fb061
spec:
  autoscaler: {}
  bucketLogging: {}
  cleanupPolicy: {}
  joinSecret:
    name: noobaa-remote-join-secret
    namespace: openshift-storage-client
  loadBalancerSourceSubnets: {}
  security:
    kms: {}
status:
  accounts:
    admin:
      secretRef: {}
  conditions:
  - lastHeartbeatTime: "2024-10-14T05:59:03Z"
    lastTransitionTime: "2024-10-14T05:59:03Z"
    message: Invalid system name "noobaa-remote" expected "noobaa"
    reason: InvalidSystemName
    status: Unknown
    type: Available
  - lastHeartbeatTime: "2024-10-14T05:59:03Z"
    lastTransitionTime: "2024-10-14T05:59:03Z"
    message: Invalid system name "noobaa-remote" expected "noobaa"
    reason: InvalidSystemName
    status: "False"
    type: Progressing
  - lastHeartbeatTime: "2024-10-14T05:59:03Z"
    lastTransitionTime: "2024-10-14T05:59:03Z"
    message: Invalid system name "noobaa-remote" expected "noobaa"
    reason: InvalidSystemName
    status: "True"
    type: Degraded
  - lastHeartbeatTime: "2024-10-14T05:59:03Z"
    lastTransitionTime: "2024-10-14T05:59:03Z"
    message: Invalid system name "noobaa-remote" expected "noobaa"
    reason: InvalidSystemName
    status: Unknown
    type: Upgradeable
  observedGeneration: 2
  phase: Rejected
  readme: "\n\tERROR: NooBaa operator cannot reconcile this system spec.\n\n\tCheck
    out the system status.phase, status.conditions, and events with:\n\n\t\tkubectl
    -n openshift-storage-client describe noobaa\n\t\tkubectl -n openshift-storage-client
    get noobaa -o yaml\n\t\tkubectl -n openshift-storage-client get events --sort-by=metadata.creationTimestamp\n\n\tIn
    order to retry, edit the system spec and the operator is watching and will be
    notified.\n\n\tNooBaa Core Version:     master-20240520\n\tNooBaa Operator Version:
    5.17.0\n"
  services:
    serviceMgmt: {}
    serviceS3: {}
    serviceSts: {}
    serviceSyslog: {}


Subs and CSVs are deleted using CLI. There is no installed operators in UI.


=================================================
Version-Release number of selected component (if applicable):
OCS client operator 4.17.0-120
Openshift 4.17.0-rc.0

==================================================
How reproducible:
1/1

Steps to Reproduce:

1. Delete all the PVCs that use the storageclass from the client
2. Delete the storageclaims that are manually created (if any for testing)
3. Delete the storageclient
4. Delete the configmap ocs-client-operator-config
5. Delete the subscription for ocs-client-operator, odf-csi-addons-operator, cephcsi-operator, mcg-operator
6. Delete the CSVs ocs-client-operator, odf-csi-addons-operator, cephcsi-operator, mcg-operator
7. Delete the client namespace

Adding output from the test.

% oc get pv,pvc,volumesnapshot -A
No resources found


% oc get storageclient           
NAME             PHASE       CONSUMER
storage-client   Connected   122e3c08-067e-4ccf-b5bc-26d42ff8714d

% oc delete storageclient storage-client
storageclient.ocs.openshift.io ""storage-client"" deleted

% oc get storageclient
No resources found

% oc get sc  
NAME                                   PROVISIONER       RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
kubevirt-csi-infra-default (default)   csi.kubevirt.io   Delete          Immediate           false                  120m

Cephblockpoolradosnamespace and subvolumegroup are deleted from provider cluster

Storageclient no longer listed in provider cluster UI

After storageclient is off-boarded

% oc get storageclient
No resources found

% oc delete configmap ocs-client-operator-config -n openshift-storage-client
configmap "ocs-client-operator-config" deleted

% oc get subs
NAME                                                                          PACKAGE                   SOURCE              CHANNEL
cephcsi-operator-stable-4.17-ocs-catalogsource-openshift-marketplace          cephcsi-operator          ocs-catalogsource   stable-4.17
mcg-operator-stable-4.17-ocs-catalogsource-openshift-marketplace              mcg-operator              ocs-catalogsource   stable-4.17
ocs-client-operator                                                           ocs-client-operator       ocs-catalogsource   stable-4.17
odf-csi-addons-operator-stable-4.17-ocs-catalogsource-openshift-marketplace   odf-csi-addons-operator   ocs-catalogsource   stable-4.17

% oc delete subs  ocs-client-operator -n openshift-storage-client
subscription.operators.coreos.com ""ocs-client-operator"" deleted

% oc delete subs odf-csi-addons-operator-stable-4.17-ocs-catalogsource-openshift-marketplace -n openshift-storage-client
subscription.operators.coreos.com ""odf-csi-addons-operator-stable-4.17-ocs-catalogsource-openshift-marketplace"" deleted

% oc delete subs cephcsi-operator-stable-4.17-ocs-catalogsource-openshift-marketplace -n openshift-storage-client
subscription.operators.coreos.com ""cephcsi-operator-stable-4.17-ocs-catalogsource-openshift-marketplace"" deleted

% oc delete subs mcg-operator-stable-4.17-ocs-catalogsource-openshift-marketplace -n openshift-storage-client
subscription.operators.coreos.com ""mcg-operator-stable-4.17-ocs-catalogsource-openshift-marketplace"" deleted

% oc get subs
No resources found in openshift-storage-client namespace.

% oc get csv 
NAME                                         DISPLAY                            VERSION             REPLACES   PHASE
cephcsi-operator.v4.17.0-120.stable          CephCSI operator                   4.17.0-120.stable              Succeeded
mcg-operator.v4.17.0-120.stable              NooBaa Operator                    4.17.0-120.stable              Succeeded
ocs-client-operator.v4.17.0-120.stable       OpenShift Data Foundation Client   4.17.0-120.stable              Succeeded
odf-csi-addons-operator.v4.17.0-120.stable   CSI Addons                         4.17.0-120.stable              Succeeded

% oc delete csv ocs-client-operator.v4.17.0-120.stable
clusterserviceversion.operators.coreos.com ""ocs-client-operator.v4.17.0-120.stable"" deleted

% oc delete csv odf-csi-addons-operator.v4.17.0-120.stable cephcsi-operator.v4.17.0-120.stable mcg-operator.v4.17.0-120.stable
clusterserviceversion.operators.coreos.com ""odf-csi-addons-operator.v4.17.0-120.stable"" deleted
clusterserviceversion.operators.coreos.com ""cephcsi-operator.v4.17.0-120.stable"" del

% oc get csv
No resources found in openshift-storage-client namespace.

All operators are gone from installed Operators page in client cluster UI"

Delete the namespace where the client operator was installed.

% oc delete namespace openshift-storage-client
namespace "openshift-storage-client" deleted
^C%    


============================================

Actual results:
Namespace openshift-storage-client is not deleted.

Expected results:
Namespace openshift-storage-client should be deleted.

Additional info:


Note You need to log in before you can comment on or make changes to this bug.