Bug 2062990

Description of problem (please be detailed as possible and provide log
snippests):
========================================================================
This bug is raised to track the logging changes needed when provider opt-out due to AllowRemoteStorageConsumers=false waits for storageconsumer deletion 

We have 2 ways to opt-out as a provider
- delete the storageCluster
- disable the AllowRemoteStorageConsumers flag and keep the storagecluster


As seen in Bug 2057001#c10, even if one set allowRemoteConsumers=false, the proivder opt-out is stuck if storageconsumers exist, but the log message says "Failed to uninstall storage cluster" . We are not trying to delete the storagecluster on provider here, just opting out of "provider" config here.

As commented here Bug 2057001#c14, "Actually, The same msg is returned if storage consumers are present in both cases, we can change the message entirely which makes sense in both cases."

Message in oc describe and ocs operator logs
=================================================
"Error while reconciling: Failed to uninstall storage cluster. StorageConsumers are present in the openshift-storage namespace. Clean all StorageConsumers for the uninstallation to proceed
"

Version of all relevant components (if applicable):
=======================================================
ODF =4.10.0-184.ci
OCP =4.9.23

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?
=========================================================================
No

Is there any workaround available to the best of your knowledge?
===================================================================
NA

Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?
===============================================================================2


Can this issue reproducible?
==================================
Yes


Can this issue reproduce from the UI?
=========================================
NA

If this is a regression, please provide more details to justify this:
=======================================================================
It is a new feature in 4.10


Steps to Reproduce:
==========================

----------------------------
1. created a odf to odf provider cluster with ODF 4.10.0-184.ci
2. onboarded the consumer and make sure storageconsumer resource exists
3. disable the AllowRemoteStorageConsumers flag in the storagecluster in the provider cluster.

# oc edit storagecluster -n openshift-storage
Spec:
  Allow Remote Storage Consumers:  false



Actual results:
===================

Observations
=================
"Error while reconciling: Failed to uninstall storage cluster. StorageConsumers are present in the openshift-storage namespace. Clean all StorageConsumers for the uninstallation to proceed
"

# oc describe storagecluster

ocs-operator is making sure storageConsumer does not exist.

Status:
  Conditions:
    Last Heartbeat Time:   2022-03-10T20:15:08Z
    Last Transition Time:  2022-03-10T20:15:03Z
    Message:               Error while reconciling: Failed to uninstall storage cluster. StorageConsumers are present in the openshift-storage namespace. Clean all StorageConsumers for the uninstallation to proceed


{"level":"info","ts":1646943701.7139826,"logger":"controllers.StorageCluster","msg":"Spec.AllowRemoteStorageConsumers is disabled","Request.Namespace":"openshift-storage","Request.Name":"ocs-storagecluster"}
{"level":"error","ts":1646943701.71401,"logger":"controllers.StorageCluster","msg":"Uninstall: Waiting for StorageConsumers to be deleted via user.","Request.Namespace":"openshift-storage","Request.Name":"ocs-storagecluster","error":"Failed to uninstall storage cluster. StorageConsumers are present in the openshift-storage namespace. Clean all StorageConsumers for the uninstallation to proceed","stacktrace":"github.com/red-hat-storage/ocs-operator/controllers/storagecluster.(*ocsProviderServer).ensureCreated\n\t/remote-source/app/controllers/storagecluster/provider_server.go:41\ngithub.com/red-hat-storage/ocs-operator/controllers/storagecluster.(*StorageClusterReconciler).reconcilePhases\n\t/remote-source/app/controllers/storagecluster/reconcile.go:401\ngithub.com/red-hat-storage/ocs-operator/controllers/storagecluster.(*StorageClusterReconciler).Reconcile\n\t/remote-source/app/controllers/storagecluster/reconcile.go:161\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214"}
{"level":"error","ts":1646943701.7310047,"logger":"controller-runtime.manager.controller.storagecluster","msg":"Reconciler error","reconciler group":"ocs.openshift.io","reconciler kind":"StorageCluster","name":"ocs-storagecluster","namespace":"openshift-storage","error":"Failed to uninstall storage cluster. StorageConsumers are present in the openshift-storage namespace. Clean all StorageConsumers for the uninstallation to proceed","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/remote-source/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214"}



Expected results:
============================
The message in logs for the opting out due to allowRemoteConsumers=false should be different than deleting storagecluster

Additional info:
====================
Confirmed with Nitin :

if I add "alloeRemoteConsumers=false", the ODF storagecluster wont uninstall on deleting the storageconsumer