Bug 1860431

Description of problem (please be detailed as possible and provide log
snippests):
----------------------------------------------------------------------
As per https://bugzilla.redhat.com/show_bug.cgi?id=1849532#c5, deleting a storagecluster patched with cleanup.ocs.openshift.io=yes-really-destroy-data would take care of the following:

1. cleanup the cluster namespace on the dataDirHostPath
2. Delete all the ceph monitor directories on the dataDirHostPath. For example mon-a, mon-b, etc.
3. Clean up the devices on each node.

For a VMware dynamic Provisioned cluster, following were the observations:

>> #1.: The cluster namespace on the dataDirHostPath is NOT cleaned and still exists(along with files)
 
 --snip-- from the OCS nodes:


 Starting pod/compute-0-debug ...
To use host binaries, run `chroot /host`

>>ls -l /var/lib/rook/openshift-storage/

total 8
-rw-r--r--. 1 root root 152 Jul 22 10:59 client.admin.keyring
drwxr-xr-x. 3  167  167  20 Jul 22 10:59 crash
drwxr-xr-x. 3  167  167  65 Jul 22 10:59 log
-rw-r--r--. 1 root root 404 Jul 22 10:59 openshift-storage.config


#2 and #3 worked for Dynamic Provisioned cluster as the MONs and OSDs are on thin SC based devices and get deleted on deleting the namespace. MON directories are not on datadirHostPath (/var/lib/rook). We need to test on BM to get the exact results for #2 and #3


Bugs for reference:  1849105 and 1849532#c5


Version of all relevant components (if applicable):
----------------------------------------------------------------------
OCS = 4.5.0-494.ci
OCP = 4.5.0-0.nightly-2020-07-22-074214
Ceph = RHCS 4.1.z1 (14.2.8-81.el8cp)


Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
----------------------------------------------------------------------
No

Is there any workaround available to the best of your knowledge?
----------------------------------------------------------------------
We would still need to document the deletion of the contents of the /var/lib/rook folder.

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
----------------------------------------------------------------------
3

Can this issue reproducible?
----------------------------------------------------------------------
tested once

Can this issue reproduce from the UI?
----------------------------------------------------------------------
Yes

If this is a regression, please provide more details to justify this:
----------------------------------------------------------------------
Uninstall has undergone changes in OCS 4.5


Steps to Reproduce:
----------------------------------------------------------------------

Doc link for reference: [1] as OCS 4.5 Uninstall is not yet ready

1. Labelled the Storagecluster with cleanup.ocs.openshift.io=yes-really-destroy-data

$  oc label -n openshift-storage storagecluster --all cleanup.ocs.openshift.io=yes-really-destroy-data
storagecluster.ocs.openshift.io/ocs-storagecluster labeled

2. Deleted all PVCs and OBCs as per current Step #3 in OCS 4.4 docs

3.  Followed the Uninstall as per https://bugzilla.redhat.com/show_bug.cgi?id=1849532#c5 and deleted the StorageCLuster from UI (the storagecluster was already patched with  cleanup.ocs.openshift.io=yes-really-destroy-data)

4. Deleted the Storagecluster from UI : Installed Operators->OCS Operator->Storage Cluster-> Delete StorageCLuster Service->OK

5. Check the /var/lib/rook content on the OCS nodes. As per Bug 1849532#c5, the cluster namespace should be cleaned automatically.





[1] - https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/4.4/html-single/deploying_openshift_container_storage/index?lb_target=preview#assembly_uninstalling-openshift-container-storage_aws-vmware

Actual results:
----------------------------------------------------------------------
Contents of /var/lib/rook are intact for a dynamic Provisioned Cluster on Vmware

Expected results:
----------------------------------------------------------------------
Content of /var/lib/rook should be cleaned up so that we can change the document to not include manual deletion of the content of the DataDirHostPath

Additional info:
----------------------------------------------------------------------

>>Following things are expected to be removed on deleting a Storagecluster:

1. Default StorageClass
2. OCS node labels
3. OCS node taints
4. cleanup the cluster namespace on the dataDirHostPath
5. Delete all the ceph monitor directories on the dataDirHostPath. For example mon-a, mon-b, etc.
6. Clean up the devices on each node.


$ for i in $(oc get nodes|grep worker |awk '{print$1}'); do oc debug node/$i -- chroot /host /bin/bash -c "ls -l /var/lib/rook/openshift-storage/" ; done ; date --utc
Starting pod/compute-0-debug ...
To use host binaries, run `chroot /host`
total 8
-rw-r--r--. 1 root root 152 Jul 22 10:59 client.admin.keyring
drwxr-xr-x. 3  167  167  20 Jul 22 10:59 crash
drwxr-xr-x. 3  167  167  65 Jul 22 10:59 log
-rw-r--r--. 1 root root 404 Jul 22 10:59 openshift-storage.config

Removing debug pod ...
Starting pod/compute-1-debug ...
To use host binaries, run `chroot /host`
total 8
-rw-r--r--. 1 root root 152 Jul 22 10:59 client.admin.keyring
drwxr-xr-x. 3  167  167  20 Jul 22 10:59 crash
drwxr-xr-x. 3  167  167  65 Jul 22 10:59 log
-rw-r--r--. 1 root root 404 Jul 22 10:59 openshift-storage.config

Removing debug pod ...
Starting pod/compute-2-debug ...
To use host binaries, run `chroot /host`
total 8
-rw-r--r--. 1 root root 152 Jul 22 10:59 client.admin.keyring
drwxr-xr-x. 3  167  167  20 Jul 22 10:59 crash
drwxr-xr-x. 3  167  167  65 Jul 22 10:59 log
-rw-r--r--. 1 root root 404 Jul 22 10:59 openshift-storage.config