Bug 2277941

Summary: [RDR][MDR] [Discovered Apps] After performing an ODF upgrade, the MCO operator reconciles the VeleroNamespaceSecretKeyRef and CACertificates fields in the Ramen Hub configmap.
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Pratik Surve <prsurve>
Component: odf-drAssignee: Raghavendra Talur <rtalur>
odf-dr sub component: multicluster-orchestrator QA Contact: Pratik Surve <prsurve>
Status: ASSIGNED --- Docs Contact:
Severity: urgent    
Priority: unspecified CC: hnallurv, kbg, kseeger, muagarwa, rtalur, sheggodu, srangana
Version: 4.16Keywords: TestBlocker
Target Milestone: ---Flags: sheggodu: needinfo? (rtalur)
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Known Issue
Doc Text:
.MCO operator reconciles the VeleroNamespaceSecretKeyRef and CACertificates fields When the OpenShift Data Foundation operator is upgraded, the `CACertificates` and `veleroNamespaceSecretKeyRef` fields under `s3StoreProfiles` in the Ramen config are lost. Workaround: If the Ramen config had custom values for the `CACertificates` and `veleroNamespaceSecretKeyRef` fields, then set those custom values after the upgrade is performed.
 Story Points: ---
Clone Of: Environment:
Last Closed: Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2260844, 2281703    

Description Pratik Surve 2024-04-30 08:11:31 UTC 
Description of problem (please be detailed as possible and provide log
snippests):

[RDR][MDR] [Discovered Apps] After performing an ODF upgrade, the MCO operator reconciles the VeleroNamespaceSecretKeyRef and CACertificates fields in the Ramen Hub configmap.

Version of all relevant components (if applicable):

OCP version:- 4.16.0-0.nightly-2024-04-26-145258
ODF version:- 4.16.0-90
CEPH version:- ceph version 18.2.1-136.el9cp (e7edde2b655d0dd9f860dda675f9d7954f07e6e3) reef (stable)
ACM version:- 2.10.2
SUBMARINER version:- v0.17.0
VOLSYNC version:- volsync-product.v0.9.0
VOLSYNC method:- destinationCopyMethod: Direct
Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?
yes

Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1.Deploy RDR cluster
2.Add veleroNamespaceSecretKeyRef and CACertificates 
3.Perform ODF upgrade
4. check ramen hub configmap values are removed


Actual results:

$oc get cm ramen-hub-operator-config -n openshift-operators -o yaml |yq -r '.data."ramen_manager_config.yaml"'
apiVersion: ramendr.openshift.io/v1alpha1
drClusterOperator:
  catalogSourceName: redhat-operators
  catalogSourceNamespaceName: openshift-marketplace
  channelName: stable-4.16
  clusterServiceVersionName: odr-cluster-operator.v4.16.0-90.stable
  deploymentAutomationEnabled: true
  namespaceName: openshift-dr-system
  packageName: odr-cluster-operator
  s3SecretDistributionEnabled: true
health:
  healthProbeBindAddress: :8081
kind: RamenConfig
kubeObjectProtection:
  veleroNamespaceName: openshift-adp
leaderElection:
  leaderElect: true
  leaseDuration: 0s
  renewDeadline: 0s
  resourceLock: ""
  resourceName: hub.ramendr.openshift.io
  resourceNamespace: ""
  retryPeriod: 0s
maxConcurrentReconciles: 50
metrics:
  bindAddress: 127.0.0.1:9289
multiNamespace:
  FeatureEnabled: true
ramenControllerType: dr-hub
ramenOpsNamespace: openshift-dr-ops
s3StoreProfiles:
- s3Bucket: odrbucket-5ea96318004d
  s3CompatibleEndpoint: https://s3-openshift-storage.apps.prsurve-vm-d.qe.rh-ocs.com
  s3ProfileName: s3profile-prsurve-vm-d-ocs-storagecluster
  s3Region: noobaa
  s3SecretRef:
    name: c0fdb10334d3fb2593dc29c4dd08d232436af1f
- s3Bucket: odrbucket-5ea96318004d
  s3CompatibleEndpoint: https://s3-openshift-storage.apps.prsurve-c1.qe.rh-ocs.com
  s3ProfileName: s3profile-prsurve-c1-ocs-storagecluster
  s3Region: noobaa
  s3SecretRef:
    name: 70fab2734107b2d7a50aee6a6a10d2de07a91ad
volSync:
  destinationCopyMethod: Direct
webhook:
  port: 9443


Expected results:
======Before upgrade=========


apiVersion: v1
data:
  ramen_manager_config.yaml: |
    apiVersion: ramendr.openshift.io/v1alpha1
    drClusterOperator:
      catalogSourceName: redhat-operators
      catalogSourceNamespaceName: openshift-marketplace
      channelName: stable-4.16
      clusterServiceVersionName: odr-cluster-operator.v4.16.0-84.stable
      deploymentAutomationEnabled: true
      namespaceName: openshift-dr-system
      packageName: odr-cluster-operator
      s3SecretDistributionEnabled: true
    health:
      healthProbeBindAddress: :8081
    kind: RamenConfig
    kubeObjectProtection:
      veleroNamespaceName: openshift-adp
    leaderElection:
      leaderElect: true
      leaseDuration: 0s
      renewDeadline: 0s
      resourceLock: ""
      resourceName: hub.ramendr.openshift.io
      resourceNamespace: ""
      retryPeriod: 0s
    maxConcurrentReconciles: 50
    metrics:
      bindAddress: 127.0.0.1:9289
    multiNamespace:
      FeatureEnabled: true
    ramenControllerType: dr-hub
    ramenOpsNamespace: ramen-ops
    s3StoreProfiles:
    - s3Bucket: odrbucket-5ea96318004d
      s3CompatibleEndpoint: https://s3-openshift-storage.apps.prsurve-c1.qe.rh-ocs.com
      s3ProfileName: s3profile-prsurve-c1-ocs-storagecluster
      s3Region: noobaa
      s3SecretRef:
        name: 70fab2734107b2d7a50aee6a6a10d2de07a91ad
      veleroNamespaceSecretKeyRef:
        key: cloud
        name: 70fab2734107b2d7a50aee6a6a10d2de07a91ad
      CACertificates: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR1ekNDQXFPZ0F3SUJBZ0lVZEM5R2MwYkhTTGx6UVJzV2Jkam5zL1ZjWExzd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2JURWZNQjBHQTFVRUNnd1dUME5USUZGRklDMGdVbVZrSUVoaGRDd2dTVzVqTGpFUE1BMEdBMVVFQ3d3RwpUME5USUZGRk1SY3dGUVlEVlFRRERBNVBRMU1nVVVVZ1VtOXZkQ0JEUVRFZ01CNEdDU3FHU0liM0RRRUpBUllSCmIyTnpMWEZsUUhKbFpHaGhkQzVqYjIwd0hoY05NakV3T1RFME1UQTBNekExV2hjTk16RXdPVEV5TVRBME16QTEKV2pCdE1SOHdIUVlEVlFRS0RCWlBRMU1nVVVVZ0xTQlNaV1FnU0dGMExDQkpibU11TVE4d0RRWURWUVFMREFaUApRMU1nVVVVeEZ6QVZCZ05WQkFNTURrOURVeUJSUlNCU2IyOTBJRU5CTVNBd0hnWUpLb1pJaHZjTkFRa0JGaEZ2ClkzTXRjV1ZBY21Wa2FHRjBMbU52YlRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUIKQU9sNWdpOFlNUStWMVhFV2lSajlNMEpaWU9sR0lBM3NxaHR6amxFSTRJbmVHMndLOHE5K0htVmJmM29jd2F3Vwo5YmNLQWQxREk5T3YwaTNXVmU3bThMOVY5R3VNcW5DZHJ4VWtjdm8zV1AybTdQazFmQVBjNVlLOVBpWTFHUDYvCk8yVko4eGRZZDNKZjFmMzVLUlg0aUlLRlVjTENiYkExQ0FZdjQ1NklxT2RKSWM4RUdONVE5ZWdzU2FJREJsbTEKbDMrMGJTT25OVXlEWXdCbG5EZWFsVzFZUWEwZEZwTnFUdHdHeHI3K0dXdWNXcXd0ZStvNVZBUEFVR2dLcWUrKwoveW5xVzZwcTJNb0lZbEw5MDduekxBRXlkakxFV3dUSnRXSFhIbFFHVUdDV3Bod3dJN3dUZ1F4ampoMnVWNkRBCjQvcExsWkpWN2hhbFViMnFsdlBzcHZrQ0F3RUFBYU5UTUZFd0hRWURWUjBPQkJZRUZNMk1VNmdDdlo4T0p5RUYKM08zckhnWG5zRXMyTUI4R0ExVWRJd1FZTUJhQUZNMk1VNmdDdlo4T0p5RUYzTzNySGdYbnNFczJNQThHQTFVZApFd0VCL3dRRk1BTUJBZjh3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQU5BZnBSckJvcCt1VDhsaEFPL28vQXVzCkN0NkM0N0hjcGlzU0RNUmNBN3hYaHlDbXNkRXJsakpiYXU1RGVyR1B5dVBaMEhlZjZYSFovY2JLZzRUOEZQRDcKZUorVEdHdGZ5dExvRnNqQ1VmZ0JqYVRDZlNBQXBJK1Z3L3ZtUzRVS1U2QVAxcFhOUkh3TEJZYUw1Y1JXSEczNQpLZmN4NFowM0tCU2xFQUJFL2NUMkFUaC9vOXRUdXVCQjBabzdWWmlRcnB2QzBCWlVXV3Y3eE1kemtlTVA0cGZOCnFaODNUTitmWWZodWF2SE1uYncyZzBqZGNsakEwdjM5QmhJZGlKcnNaSm4vY1dhM1VuOGtHeFd1SEZMRHkwUWcKQjBiSGRRclhQUzRiNjg3VlRkM1RxSENGUXMwc1lJQ25OdFVva1NHSExlUS85MldoKzZyK2w2TGxpbVd2em5nPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    - s3Bucket: odrbucket-5ea96318004d
      s3CompatibleEndpoint: https://s3-openshift-storage.apps.prsurve-vm-d.qe.rh-ocs.com
      s3ProfileName: s3profile-prsurve-vm-d-ocs-storagecluster
      s3Region: noobaa
      s3SecretRef:
        name: c0fdb10334d3fb2593dc29c4dd08d232436af1f
      veleroNamespaceSecretKeyRef:
        key: cloud
        name: c0fdb10334d3fb2593dc29c4dd08d232436af1f
      CACertificates: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR1ekNDQXFPZ0F3SUJBZ0lVZEM5R2MwYkhTTGx6UVJzV2Jkam5zL1ZjWExzd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2JURWZNQjBHQTFVRUNnd1dUME5USUZGRklDMGdVbVZrSUVoaGRDd2dTVzVqTGpFUE1BMEdBMVVFQ3d3RwpUME5USUZGRk1SY3dGUVlEVlFRRERBNVBRMU1nVVVVZ1VtOXZkQ0JEUVRFZ01CNEdDU3FHU0liM0RRRUpBUllSCmIyTnpMWEZsUUhKbFpHaGhkQzVqYjIwd0hoY05NakV3T1RFME1UQTBNekExV2hjTk16RXdPVEV5TVRBME16QTEKV2pCdE1SOHdIUVlEVlFRS0RCWlBRMU1nVVVVZ0xTQlNaV1FnU0dGMExDQkpibU11TVE4d0RRWURWUVFMREFaUApRMU1nVVVVeEZ6QVZCZ05WQkFNTURrOURVeUJSUlNCU2IyOTBJRU5CTVNBd0hnWUpLb1pJaHZjTkFRa0JGaEZ2ClkzTXRjV1ZBY21Wa2FHRjBMbU52YlRDQ0FTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUIKQU9sNWdpOFlNUStWMVhFV2lSajlNMEpaWU9sR0lBM3NxaHR6amxFSTRJbmVHMndLOHE5K0htVmJmM29jd2F3Vwo5YmNLQWQxREk5T3YwaTNXVmU3bThMOVY5R3VNcW5DZHJ4VWtjdm8zV1AybTdQazFmQVBjNVlLOVBpWTFHUDYvCk8yVko4eGRZZDNKZjFmMzVLUlg0aUlLRlVjTENiYkExQ0FZdjQ1NklxT2RKSWM4RUdONVE5ZWdzU2FJREJsbTEKbDMrMGJTT25OVXlEWXdCbG5EZWFsVzFZUWEwZEZwTnFUdHdHeHI3K0dXdWNXcXd0ZStvNVZBUEFVR2dLcWUrKwoveW5xVzZwcTJNb0lZbEw5MDduekxBRXlkakxFV3dUSnRXSFhIbFFHVUdDV3Bod3dJN3dUZ1F4ampoMnVWNkRBCjQvcExsWkpWN2hhbFViMnFsdlBzcHZrQ0F3RUFBYU5UTUZFd0hRWURWUjBPQkJZRUZNMk1VNmdDdlo4T0p5RUYKM08zckhnWG5zRXMyTUI4R0ExVWRJd1FZTUJhQUZNMk1VNmdDdlo4T0p5RUYzTzNySGdYbnNFczJNQThHQTFVZApFd0VCL3dRRk1BTUJBZjh3RFFZSktvWklodmNOQVFFTEJRQURnZ0VCQU5BZnBSckJvcCt1VDhsaEFPL28vQXVzCkN0NkM0N0hjcGlzU0RNUmNBN3hYaHlDbXNkRXJsakpiYXU1RGVyR1B5dVBaMEhlZjZYSFovY2JLZzRUOEZQRDcKZUorVEdHdGZ5dExvRnNqQ1VmZ0JqYVRDZlNBQXBJK1Z3L3ZtUzRVS1U2QVAxcFhOUkh3TEJZYUw1Y1JXSEczNQpLZmN4NFowM0tCU2xFQUJFL2NUMkFUaC9vOXRUdXVCQjBabzdWWmlRcnB2QzBCWlVXV3Y3eE1kemtlTVA0cGZOCnFaODNUTitmWWZodWF2SE1uYncyZzBqZGNsakEwdjM5QmhJZGlKcnNaSm4vY1dhM1VuOGtHeFd1SEZMRHkwUWcKQjBiSGRRclhQUzRiNjg3VlRkM1RxSENGUXMwc1lJQ25OdFVva1NHSExlUS85MldoKzZyK2w2TGxpbVd2em5nPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==
    volSync:
      destinationCopyMethod: Direct
    webhook:
      port: 9443
kind: ConfigMap
metadata:
  creationTimestamp: "2024-04-16T05:06:21Z"
  labels:
    cluster.open-cluster-management.io/backup: resource
    olm.managed: "true"
    operators.coreos.com/odr-hub-operator.openshift-operators: ""
  name: ramen-hub-operator-config
  namespace: openshift-operators
  ownerReferences:
    - apiVersion: operators.coreos.com/v1alpha1
      blockOwnerDeletion: false
      controller: false
      kind: ClusterServiceVersion
      name: odr-hub-operator.v4.16.0-84.stable
      uid: f835d16e-e0ed-4b1b-8ce1-197ad606409a
  resourceVersion: "10062519"
  uid: cc85c38f-f780-4f98-ab3d-6bcf3410b307

Additional info: