Bug 2071923

Summary: toolbox pod and smtp secret issues with deployer-2.0.0-8 and ODF 4.10.0-219 build(ocs-provider-qe and ocs-consumer-qe)
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Neha Berry <nberry>
Component: odf-managed-serviceAssignee: Ohad <omitrani>
Status: CLOSED CURRENTRELEASE QA Contact: Neha Berry <nberry>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.10CC: aeyal, dbindra, ocs-bugs, odf-bz-bot
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-09-28 06:49:06 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
provider-add-on install none

Description Neha Berry 2022-04-05 08:43:37 UTC 
Created attachment 1870818 [details]
provider-add-on install

Description of problem:
===========================
Recently the name of the provider qe addon was changed.
Since then, facing issues with deployment of add-on :

1. smtp secret uses wrong name and toolbox pod fails to come up

Chat details - 
https://chat.google.com/room/AAAASHA9vWs/lvl_LKqt2LA
 


2022-04-05T06:40:33.144Z        ERROR        controllers.ManagedOCS        An error was encountered during reconcilePhases        {"error": "Unable to get SMTP secret: Secret \"ocs-provider-qe-smtp\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /tmp/go/ocs-osd-deployer/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:298
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /tmp/go/ocs-osd-deployer/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /tmp/go/ocs-osd-deployer/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214
2022-04-05T06:40:33.150Z        ERROR        controller-runtime.manager.controller.managedocs        Reconciler error        {"reconciler group": "ocs.openshift.io", "reconciler kind": "ManagedOCS", "name": "managedocs", "namespace": "openshift-storage", "error": "Unable to get SMTP secret: Secret \"ocs-provider-qe-smtp\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /tmp/go/ocs-osd-deployer/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /tmp/go/ocs-osd-deployer/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214


oc get secret|grep smtp
ocs-converged-qe-smtp                                       Opaque                                5      23m


Version-Release number of selected component (if applicable):
=============================================================
oc describe csv ocs-osd-deployer.v2.0.0 |grep -i image                  
    Mediatype:   image/svg+xml
                Image:  quay.io/openshift/origin-kube-rbac-proxy:4.10.0
                Image:             quay.io/osd-addons/ocs-osd-deployer:2.0.0-8
                Image:             quay.io/osd-addons/ocs-osd-deployer:2.0.0-8
➜  p oc get csv -n openshift-storage -o json ocs-operator.v4.10.0 | jq '.metadata.labels["full_version"]'     
"4.10.0-219"
➜  p oc describe catsrc redhat-operators |grep -i image    
        f:image:
  Image:         quay.io/rhceph-dev/odf4-odf-compose-operator-bundle-index:4.10.0-219

How reproducible:
=====================
Always


Steps to Reproduce:
===================
1. Install ROSA cluster - provider Openshift cluster
2. Add the security group for ceph pods
3. Install ocs-provider-qe add-on from OCM UI

Actual results:
=================
Even though cephcluster is in Health_OK, the add-on stays in Installing state for long and the manager pod shows smtp secret related issues

Expected results:
==================
No errors in deployment

Additional info:
=====================
Comment 2 Dhruv Bindra 2022-04-05 10:13:14 UTC 
The issue was that the SMTP service that integrates with OCM uses the Addon name and not the Addon ID. We have recently changed the addon names as per the official documentation which made the Addon name of converged offering similar to the addon name for provider addon. This caused the issue and the secret ocs-converged-qe-smtp was getting created for the provider-qe addon.
Comment 3 Dhruv Bindra 2022-04-05 17:02:42 UTC 
As the addon name and SMTP service config was updated, the changes needs to be tested on a new cluster. I verified changes on the new cluster and everything seems to be working fine.