Bug 2248958

Summary: [GSS] rook-ceph-operator CreateContainerConfigError ocs-operator-config
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: khover
Component: rookAssignee: Santosh Pillai <sapillai>
Status: CLOSED DUPLICATE QA Contact: Neha Berry <nberry>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.13CC: hnallurv, odf-bz-bot, sapillai, tnielsen
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-11-10 05:22:48 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description khover 2023-11-09 21:13:43 UTC 
Description of problem (please be detailed as possible and provide log
snippests):

rook-ceph-operator CreateContainerConfigError  

ODF 4.13 install   

NAME                                                                    DISPLAY                     VERSION        REPLACES                     PHASE
clusterserviceversion.operators.coreos.com/mcg-operator.v4.12.9-rhodf   NooBaa Operator             4.12.9-rhodf   mcg-operator.v4.12.8-rhodf   InstallReady
clusterserviceversion.operators.coreos.com/odf-operator.v4.13.4-rhodf   OpenShift Data Foundation   4.13.4-rhodf   odf-operator.v4.13.3-rhodf   Pending

NAME                                             CSV                          APPROVAL    APPROVED
installplan.operators.coreos.com/install-7l9lt   mcg-operator.v4.12.9-rhodf   Automatic   true

NAME                                                                                                           PACKAGE                   SOURCE             CHANNEL
subscription.operators.coreos.com/mcg-operator-stable-4.12-redhat-operators-openshift-marketplace              mcg-operator              redhat-operators   stable-4.12
subscription.operators.coreos.com/ocs-operator-stable-4.12-redhat-operators-openshift-marketplace              ocs-operator              redhat-operators   stable-4.12
subscription.operators.coreos.com/odf-csi-addons-operator-stable-4.12-redhat-operators-openshift-marketplace   odf-csi-addons-operator   redhat-operators   stable-4.12
subscription.operators.coreos.com/odf-operator                                                                 odf-operator              redhat-operators   stable-4.13


2

NAME                                                                               DISPLAY                       VERSION        REPLACES                                PHASE
clusterserviceversion.operators.coreos.com/mcg-operator.v4.12.9-rhodf              NooBaa Operator               4.12.9-rhodf   mcg-operator.v4.12.8-rhodf              Replacing
clusterserviceversion.operators.coreos.com/mcg-operator.v4.13.4-rhodf              NooBaa Operator               4.13.4-rhodf   mcg-operator.v4.12.9-rhodf              Succeeded
clusterserviceversion.operators.coreos.com/ocs-operator.v4.12.9-rhodf              OpenShift Container Storage   4.12.9-rhodf   ocs-operator.v4.12.8-rhodf              Replacing
clusterserviceversion.operators.coreos.com/ocs-operator.v4.13.4-rhodf              OpenShift Container Storage   4.13.4-rhodf   ocs-operator.v4.12.9-rhodf              Pending
clusterserviceversion.operators.coreos.com/odf-csi-addons-operator.v4.12.9-rhodf   CSI Addons                    4.12.9-rhodf   odf-csi-addons-operator.v4.12.8-rhodf   Succeeded
clusterserviceversion.operators.coreos.com/odf-operator.v4.13.4-rhodf              OpenShift Data Foundation     4.13.4-rhodf   odf-operator.v4.13.3-rhodf              Succeeded


**Stuck installing 

NAME                                                                               DISPLAY                       VERSION        REPLACES                                PHASE
clusterserviceversion.operators.coreos.com/mcg-operator.v4.13.4-rhodf              NooBaa Operator               4.13.4-rhodf   mcg-operator.v4.12.9-rhodf              Succeeded
clusterserviceversion.operators.coreos.com/ocs-operator.v4.12.9-rhodf              OpenShift Container Storage   4.12.9-rhodf   ocs-operator.v4.12.8-rhodf              Replacing
clusterserviceversion.operators.coreos.com/ocs-operator.v4.13.4-rhodf              OpenShift Container Storage   4.13.4-rhodf   ocs-operator.v4.12.9-rhodf              Installing
clusterserviceversion.operators.coreos.com/odf-csi-addons-operator.v4.13.4-rhodf   CSI Addons                    4.13.4-rhodf   odf-csi-addons-operator.v4.12.9-rhodf   Succeeded
clusterserviceversion.operators.coreos.com/odf-operator.v4.13.4-rhodf              OpenShift Data Foundation     4.13.4-rhodf   odf-operator.v4.13.3-rhodf              Succeeded

**Same error

pod/rook-ceph-operator-dd6c49498-2rpqw                        Error: couldn't find key CSI_ENABLE_READ_AFFINITY in ConfigMap openshift-storage/ocs-operator-config


Version of all relevant components (if applicable):

4.13.4

Does this issue impact your ability to continue to work with the product
(please explain in detail what is the user impact)?


Is there any workaround available to the best of your knowledge?


Rate from 1 - 5 the complexity of the scenario you performed that caused this
bug (1 - very simple, 5 - very complex)?


Can this issue reproducible?

Yes, verified in lab and on customer environment.


Can this issue reproduce from the UI?


If this is a regression, please provide more details to justify this:


Steps to Reproduce:
1. Install ODF 4.13 
2.
3.


Actual results:

Error: couldn't find key CSI_ENABLE_READ_AFFINITY in ConfigMap openshift-storage/ocs-operator-config

Expected results:

Configs applied to ocs-operator-config.




Additional info:
Comment 2 khover 2023-11-09 21:28:20 UTC 
Only known work around is to add the configs manually to cm ocs-operator-config
Comment 3 khover 2023-11-09 21:34:29 UTC 
Also observed and not sure if this is by design.

When installing ODF 4.13 it seems to start at 4.12.8 then upgrade to 4.12.9 then 4.13.4

NAME                                                                               DISPLAY                       VERSION        REPLACES                                PHASE
clusterserviceversion.operators.coreos.com/mcg-operator.v4.13.4-rhodf              NooBaa Operator               4.13.4-rhodf   mcg-operator.v4.12.9-rhodf              Succeeded
clusterserviceversion.operators.coreos.com/ocs-operator.v4.12.9-rhodf              OpenShift Container Storage   4.12.9-rhodf   ocs-operator.v4.12.8-rhodf              Replacing
clusterserviceversion.operators.coreos.com/ocs-operator.v4.13.4-rhodf              OpenShift Container Storage   4.13.4-rhodf   ocs-operator.v4.12.9-rhodf              Failed
clusterserviceversion.operators.coreos.com/odf-csi-addons-operator.v4.13.4-rhodf   CSI Addons                    4.13.4-rhodf   odf-csi-addons-operator.v4.12.9-rhodf   Succeeded
clusterserviceversion.operators.coreos.com/odf-operator.v4.13.4-rhodf              OpenShift Data Foundation     4.13.4-rhodf   odf-operator.v4.13.3-rhodf              Succeeded


When installing 4.12 it starts at 4.12.8 then completes on 4.12.9
Comment 4 Santosh Pillai 2023-11-10 05:13:59 UTC 
looks like same as https://bugzilla.redhat.com/show_bug.cgi?id=2231074
Comment 5 Santosh Pillai 2023-11-10 05:22:48 UTC 
closing it as a duplicate https://bugzilla.redhat.com/show_bug.cgi?id=2235571 which is a backport PR for the fix in 4.13. Currently targeted for 4.13.6

*** This bug has been marked as a duplicate of bug 2235571 ***
Comment 6 Red Hat Bugzilla 2024-03-20 04:25:09 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days