Bug 2276855

Summary: [Non-provider mode] uninstallation is stuck in deleting openshift-storage NS due to ocs-client-operator.ocs.openshift.io/storageused
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Vijay Avuthu <vavuthu>
Component: documentationAssignee: Kusuma <kbg>
Status: VERIFIED --- QA Contact: Erin Donnelly <edonnell>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.16CC: aclewett, lgangava, mrajanna, muagarwa, nberry, odf-bz-bot, omitrani, rchikatw, rohgupta
Target Milestone: ---Keywords: NoDocsQEReview
Target Release: ---Flags: kbg: needinfo-
Hardware: Unspecified   
OS: Unspecified   
Whiteboard: isf-provider
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Vijay Avuthu 2024-04-24 10:03:55 UTC 
Description of problem:
Uninstallation of ODF is stuck in deleting namespace "openshift-storage" due to 'Some content in the namespace has finalizers remaining: ocs-client-operator.ocs.openshift.io/storageused in 1 resource instances'


Version-Release number of selected component (if applicable):
ocs-registry:4.16.0-84

How reproducible:
1/1

Steps to Reproduce:
1. uninstall ODF using KCS https://access.redhat.com/articles/6525111
2.
3.

Actual results:

step 5, still contains entries

$ for i in $(oc get node -l cluster.ocs.openshift.io/openshift-storage= -o jsonpath='{ .items[*].metadata.name  }'); do oc debug node/${i} -- chroot /host  ls -l /var/lib/rook; done

Starting pod/compute-0-debug-4c2qj ...
To use host binaries, run `chroot /host`
total 0
drwxr-xr-x. 2  167  167  67 Apr 23 12:26 exporter
drwxr-xr-x. 5 root root 125 Apr 23 07:26 openshift-storage

Removing debug pod ...
Starting pod/compute-1-debug-97k5d ...
To use host binaries, run `chroot /host`
total 0
drwxr-xr-x. 2  167  167  90 Apr 23 12:26 exporter
drwxr-xr-x. 5 root root 125 Apr 23 07:26 openshift-storage

Removing debug pod ...
Starting pod/compute-2-debug-42h7k ...
To use host binaries, run `chroot /host`
total 0
drwxr-xr-x. 2  167  167  90 Apr 23 12:26 exporter
drwxr-xr-x. 5 root root 125 Apr 23 07:26 openshift-storage

Removing debug pod ...
$ 

After step 7 , openshift-storage is in terminating stage.

$ oc get project openshift-storage

NAME                DISPLAY NAME   STATUS
openshift-storage                  Terminating

Expected results:

uninstallation should complete without any issues

Additional info:


$ oc get project openshift-storage -o yaml
apiVersion: project.openshift.io/v1
kind: Project
metadata:
  annotations:
    openshift.io/sa.scc.mcs: s0:c26,c20
    openshift.io/sa.scc.supplemental-groups: 1000690000/10000
    openshift.io/sa.scc.uid-range: 1000690000/10000
    reclaimspace.csiaddons.openshift.io/schedule: '@weekly'
  creationTimestamp: "2024-04-23T07:19:49Z"
  deletionTimestamp: "2024-04-23T12:36:28Z"
  labels:
    kubernetes.io/metadata.name: openshift-storage
    olm.operatorgroup.uid/1b09746f-e150-4b88-bb75-fa8c5fa6d4fc: ""
    openshift.io/cluster-monitoring: "true"
    pod-security.kubernetes.io/audit: privileged
    pod-security.kubernetes.io/audit-version: v1.24
    pod-security.kubernetes.io/enforce: privileged
    pod-security.kubernetes.io/enforce-version: v1.24
    pod-security.kubernetes.io/warn: privileged
    pod-security.kubernetes.io/warn-version: v1.24
    security.openshift.io/scc.podSecurityLabelSync: "true"
  name: openshift-storage
  resourceVersion: "238385"
  uid: 1dac99b1-d7f9-4f44-8763-f25cc8b9899f
spec:
  finalizers:
  - kubernetes
status:
  conditions:
  - lastTransitionTime: "2024-04-23T12:36:42Z"
    message: All resources successfully discovered
    reason: ResourcesDiscovered
    status: "False"
    type: NamespaceDeletionDiscoveryFailure
  - lastTransitionTime: "2024-04-23T12:36:42Z"
    message: All legacy kube types successfully parsed
    reason: ParsedGroupVersions
    status: "False"
    type: NamespaceDeletionGroupVersionParsingFailure
  - lastTransitionTime: "2024-04-23T12:36:42Z"
    message: All content successfully deleted, may be waiting on finalization
    reason: ContentDeleted
    status: "False"
    type: NamespaceDeletionContentFailure
  - lastTransitionTime: "2024-04-23T12:36:42Z"
    message: 'Some resources are remaining: configmaps. has 1 resource instances'
    reason: SomeResourcesRemain
    status: "True"
    type: NamespaceContentRemaining
  - lastTransitionTime: "2024-04-23T12:36:42Z"
    message: 'Some content in the namespace has finalizers remaining: ocs-client-operator.ocs.openshift.io/storageused
      in 1 resource instances'
    reason: SomeFinalizersRemain
    status: "True"
    type: NamespaceFinalizersRemaining
  phase: Terminating

> no resources found for StorageClient
$ oc get storageclients.ocs.openshift.io -n openshift-storage
No resources found
$

$ oc get csv -n openshift-storage
No resources found in openshift-storage namespace.
$

> request is to provide steps needed to uninstall ODF