User Story: As an OpenShift administrator using OCS 4.3, I want a user-friendly method to uninstall OCS 4.3 so that there is no data, etc., remaining. Content Plan: https://docs.google.com/document/d/1BOPHnBmZf7ApzSF6VoXsImaFVIwEuLgHvFEGEzfeSGg/edit?usp=sharing
Uninstalled OCS on VMWARE and AWS following doc https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/4.3/html-single/deploying_openshift_container_storage/index?lb_target=preview#uninstalling-openshift-container-storage_rhocs. Here are few observations: -------------------------- 1. As mentioned on step2, while deleting the pvc it hungs and pvc will be in terminating state. $ oc get pvc -o=jsonpath='{range .items[?(@.spec.storageClassName=="ocs-storagecluster-ceph-rbd")]}{"Name: "}{@.metadata.name}{" Namespace: "}{@.metadata.namespace}{"\n"}{end}' --all-namespaces|grep -v db-noobaa-core-0 Name: db-noobaa-db-0 Namespace: openshift-storage $ oc delete pvc db-noobaa-db-0 -n openshift-storage persistentvolumeclaim "db-noobaa-db-0" deleted ^Z [1]+ Stopped oc delete pvc db-noobaa-db-0 -n openshift-storage NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/db-noobaa-db-0 Terminating pvc-9cdfe325-a482-4c3f-8750-fe2d35121499 50Gi RWO ocs-storagecluster-ceph-rbd 35m persistentvolumeclaim/ocs-deviceset-0-0-flfjv Bound pvc-a93b8d95-c2d3-4053-8d67-0011d305a963 2Ti RWO thin 36m 2. On uninstall logging, all pods and pvc deleted expect cluster-logging-operator. $ date; oc get pods,pvc -n openshift-logging Fri Mar 13 17:36:07 IST 2020 NAME READY STATUS RESTARTS AGE pod/cluster-logging-operator-75cbd8c596-dcb6m 1/1 Running 0 15m 3. On step8 will unlabel the nodes and in step10 listing those nodes with label will fail. It's better to remove listing nodes with label and list only the OCS worker node. $ oc get nodes -l cluster.ocs.openshift.io/openshift-storage= No resources found. 4. For step8, good to have a ignore warning for unlabeled nodes. $ oc label nodes --all cluster.ocs.openshift.io/openshift-storage- node/compute-0 labeled node/compute-1 labeled node/compute-2 labeled label "cluster.ocs.openshift.io/openshift-storage" not found. node/control-plane-0 not labeled label "cluster.ocs.openshift.io/openshift-storage" not found. node/control-plane-1 not labeled label "cluster.ocs.openshift.io/openshift-storage" not found. node/control-plane-2 not labeled $ oc label nodes --all topology.rook.io/rack- node/compute-0 labeled node/compute-1 labeled node/compute-2 labeled label "topology.rook.io/rack" not found. node/control-plane-0 not labeled label "topology.rook.io/rack" not found. node/control-plane-1 not labeled label "topology.rook.io/rack" not found. node/control-plane-2 not labeled 5. The pods(i.e, especially ocs-catalogsource) created on namespace openshift-marketplace are not deleted. $ oc get pods -n openshift-marketplace NAME READY STATUS RESTARTS AGE certified-operators-c6bf689d7-tczcg 1/1 Running 0 48m community-operators-5dd597588d-mfkdg 1/1 Running 0 48m marketplace-operator-5d954b74fb-9hxsq 1/1 Running 0 48m ocs-catalogsource-l2s2b 1/1 Running 0 36m redhat-operators-6f59c9f797-2vz25 1/1 Running 0 48m Based on above observation moving the bz to failed_qa.
Looks like the name of the NooBaa PV we're supposed to ignore has changed to db-noobaa-db-0. This looks like the result of deleting the NooBaa PV. Looks like it's name changed and it's no longer omitted by the query. I think it will be best to document with the queries "ignore any NooBaa PVCs in the openshift-storage namespace"
Didn't address everything before: 1. We don't want to delete the NooBaa pvc, so we should document with the queries "ignore any NooBaa PVCs in the openshift-storage namespace" Also, I'll provide a better query in my next comment. 2. Looks fine. 3. Let's move steps 10,11 to just after step 5 (deleting the project). 4. Good point! Let's document: Ignore 'label "<label>" not found.' warnings for these steps. 5. ocs-catalogsource is not going to be present on production environments as they install from OperatorHub
Needinfoing myself so I don't forget to add the query
The new RBD query: oc get pvc -o=jsonpath='{range .items[?(@.spec.storageClassName=="ocs-storagecluster-ceph-rbd")]}{"Name: "}{@.metadata.name}{" Namespace: "}{@.metadata.namespace}{" Labels: "}{@.metadata.labels}{"\n"}{end}' --all-namespaces|awk '! ( /Namespace: openshift-storage/ && /app:noobaa/ )'
This query will ignore PVCs in the openshift-storage namespace that have the app: noobaa label. These are taken care of by the operator with a finalizer.
Following the doc https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/4.3/html-single/deploying_openshift_container_storage/index?lb_target=preview#uninstalling-openshift-container-storage_rhocs, was able to unistall and install OCS again. So moving this bz to verified. Version: -------- $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.3.8 True False 27m Cluster version is 4.3.8 $ oc get csv -n openshift-storage NAME DISPLAY VERSION REPLACES PHASE lib-bucket-provisioner.v1.0.0 lib-bucket-provisioner 1.0.0 Succeeded ocs-operator.v4.3.0-379.ci OpenShift Container Storage 4.3.0-379.ci Succeeded Steps performed: ---------------- 1. Deleted all the pvcs, pod as a part of created while testing, configuring the monitoring stack, cluster logging operator, and registry. $ oc get pvc -o=jsonpath='{range .items[?(@.spec.storageClassName=="ocs-storagecluster-ceph-rbd")]}{"Name: "}{@.metadata.name}{" Namespace: "}{@.metadata.namespace}{" Labels: "}{@.metadata.labels}{"\n"}{end}' --all-namespaces|awk '! ( /Namespace: openshift-storage/ && /app:noobaa/ )' $ oc get pvc -o=jsonpath='{range .items[?(@.spec.storageClassName=="ocs-storagecluster-cephfs")]}{"Name: "}{@.metadata.name}{" Namespace: "}{@.metadata.namespace}{"\n"}{end}' --all-namespaces $ oc get obc -o=jsonpath='{range .items[?(@.spec.storageClassName=="openshift-storage.noobaa.io")]}{"Name: "}{@.metadata.name}{" Namespace: "}{@.metadata.namespace}{"\n"}{end}' --all-namespaces 2. Deleted storagecluster $ oc delete -n openshift-storage storagecluster --all --wait=true storagecluster.ocs.openshift.io "ocs-storagecluster" deleted 3. Deleted openshift-storage project. $ oc delete project openshift-storage --wait=true --timeout=5m project.project.openshift.io "openshift-storage" deleted 4. On each ocs node, cleaned up the ocs components. $ oc get nodes -l cluster.ocs.openshift.io/openshift-storage= NAME STATUS ROLES AGE VERSION compute-0 Ready worker 71m v1.16.2 compute-1 Ready worker 71m v1.16.2 compute-2 Ready worker 71m v1.16.2 $ oc debug node/compute-0 Starting pod/compute-0-debug ... To use host binaries, run `chroot /host` Pod IP: 10.70.40.11 If you don't see a command prompt, try pressing enter. sh-4.2# chroot /host sh-4.4# rm -rf /var/lib/rook sh-4.4# exit exit sh-4.2# exit exit Removing debug pod ... $ oc debug node/compute-1 Starting pod/compute-1-debug ... To use host binaries, run `chroot /host` Pod IP: 10.70.40.12 If you don't see a command prompt, try pressing enter. sh-4.2# chroot /host sh-4.4# rm -rf /var/lib/rook sh-4.4# exit exit sh-4.2# exit exit Removing debug pod ... $ oc debug node/compute-2 Starting pod/compute-2-debug ... To use host binaries, run `chroot /host` Pod IP: 10.70.40.7 If you don't see a command prompt, try pressing enter. sh-4.2# chroot /host sh-4.4# rm -rf /var/lib/rook sh-4.4# exit exit sh-4.2# exit exit Removing debug pod ... 5. Deleted storage classes. $ oc delete storageclass ocs-storagecluster-ceph-rbd ocs-storagecluster-cephfs openshift-storage.noobaa.io --wait=true --timeout=5m storageclass.storage.k8s.io "ocs-storagecluster-ceph-rbd" deleted storageclass.storage.k8s.io "ocs-storagecluster-cephfs" deleted storageclass.storage.k8s.io "openshift-storage.noobaa.io" deleted 6. Removed taints and unlabeled nodes. $ oc label nodes --all cluster.ocs.openshift.io/openshift-storage- node/compute-0 labeled node/compute-1 labeled node/compute-2 labeled label "cluster.ocs.openshift.io/openshift-storage" not found. node/control-plane-0 not labeled label "cluster.ocs.openshift.io/openshift-storage" not found. node/control-plane-1 not labeled label "cluster.ocs.openshift.io/openshift-storage" not found. node/control-plane-2 not labeled $ oc label nodes --all topology.rook.io/rack- node/compute-0 labeled node/compute-1 labeled node/compute-2 labeled label "topology.rook.io/rack" not found. node/control-plane-0 not labeled label "topology.rook.io/rack" not found. node/control-plane-1 not labeled label "topology.rook.io/rack" not found. node/control-plane-2 not labeled 7. Deleted crd. $ oc delete crd backingstores.noobaa.io bucketclasses.noobaa.io cephblockpools.ceph.rook.io cephclusters.ceph.rook.io cephfilesystems.ceph.rook.io cephnfses.ceph.rook.io cephobjectstores.ceph.rook.io cephobjectstoreusers.ceph.rook.io noobaas.noobaa.io ocsinitializations.ocs.openshift.io storageclusterinitializations.ocs.openshift.io storageclusters.ocs.openshift.io --wait=true --timeout=5m customresourcedefinition.apiextensions.k8s.io "backingstores.noobaa.io" deleted customresourcedefinition.apiextensions.k8s.io "bucketclasses.noobaa.io" deleted customresourcedefinition.apiextensions.k8s.io "cephblockpools.ceph.rook.io" deleted customresourcedefinition.apiextensions.k8s.io "cephclusters.ceph.rook.io" deleted customresourcedefinition.apiextensions.k8s.io "cephfilesystems.ceph.rook.io" deleted customresourcedefinition.apiextensions.k8s.io "cephnfses.ceph.rook.io" deleted customresourcedefinition.apiextensions.k8s.io "cephobjectstores.ceph.rook.io" deleted customresourcedefinition.apiextensions.k8s.io "cephobjectstoreusers.ceph.rook.io" deleted customresourcedefinition.apiextensions.k8s.io "noobaas.noobaa.io" deleted customresourcedefinition.apiextensions.k8s.io "ocsinitializations.ocs.openshift.io" deleted customresourcedefinition.apiextensions.k8s.io "storageclusterinitializations.ocs.openshift.io" deleted customresourcedefinition.apiextensions.k8s.io "storageclusters.ocs.openshift.io" deleted 8. Verified openshift-storage namespace is deleted and also no storage dashboard appears on UI. 9. Installed OCS again on the cluster. All pods are up and running.