2183266 – [Fusion aaS Rook ODF 4.13]] Rook-ceph-operator pod should allow OBC CRDs to be optional instead of causing a crash when not present

Bug 2183266 - [Fusion aaS Rook ODF 4.13]] Rook-ceph-operator pod should allow OBC CRDs to be optional instead of causing a crash when not present

Summary: [Fusion aaS Rook ODF 4.13]] Rook-ceph-operator pod should allow OBC CRDs to b...

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenShift Data Foundation
Classification:	Red Hat Storage
Component:	rook
Sub Component:
Version:	4.12
Hardware:	Unspecified
OS:	Unspecified
Priority:	unspecified
Severity:	high
Target Milestone:	---
Target Release:	ODF 4.13.0
Assignee:	Santosh Pillai
QA Contact:	suchita
Docs Contact:
URL:
Whiteboard:
Depends On:	2183259
Blocks:	2187765
TreeView+	depends on / blocked

Reported:	2023-03-30 17:22 UTC by Neha Berry
Modified:	2023-12-08 04:32 UTC (History)
CC List:	12 users (show)
Fixed In Version:
Doc Type:	No Doc Update
Doc Text:
Clone Of:	2183259
Clones:	2187765 (view as bug list)
Environment:
Last Closed:	2023-06-21 15:25:01 UTC
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Priority	Status	Summary	Last Updated
Github	red-hat-storage rook pull 482	None	Merged	Bug: 2183266: core: skip OBC and Notification controllers	2023-05-02 15:24:03 UTC
Github	rook rook pull 12075	None	open	[WIP]core: Skip controllers based on env variable	2023-04-13 13:43:06 UTC
Red Hat Product Errata	RHBA-2023:3742	None	None	None	2023-06-21 15:25:29 UTC

Comment 15 Santosh Pillai 2023-04-07 07:52:35 UTC

so had a discussion with the team about possible fixes:

- OCS operator could pass on details va env/config map and provide the resources that rook should ignore and not start a watch on.
- Rook should itself check if a CRD is available before starting a watch on the resource. 

Waiting on OCS operator team to see how easy it is to pass this info to rook config map.

Comment 17 Travis Nielsen 2023-04-13 13:43:07 UTC

Neha, which CRDs exactly are being created in the cluster? We need to confirm if we are skipping all the necessary CRDs. 

What does this show?
oc get crd

Comment 18 Santosh Pillai 2023-04-14 05:45:24 UTC

must gather does not collect CRD info.

Comment 20 Filip Balák 2023-04-17 08:40:51 UTC

$ oc get crd
NAME                                                              CREATED AT
addoninstances.addons.managed.openshift.io                        2023-04-17T05:22:42Z
addonoperators.addons.managed.openshift.io                        2023-04-17T05:22:42Z
addons.addons.managed.openshift.io                                2023-04-17T05:22:42Z
alertmanagerconfigs.monitoring.coreos.com                         2023-04-17T04:53:21Z
alertmanagerconfigs.monitoring.rhobs                              2023-04-17T05:22:31Z
alertmanagers.monitoring.coreos.com                               2023-04-17T04:53:24Z
alertmanagers.monitoring.rhobs                                    2023-04-17T05:22:30Z
apirequestcounts.apiserver.openshift.io                           2023-04-17T04:53:04Z
apiservers.config.openshift.io                                    2023-04-17T04:52:39Z
authentications.config.openshift.io                               2023-04-17T04:52:40Z
authentications.operator.openshift.io                             2023-04-17T04:53:22Z
backups.velero.io                                                 2023-04-17T05:22:18Z
backupstoragelocations.velero.io                                  2023-04-17T05:22:18Z
baremetalhosts.metal3.io                                          2023-04-17T04:53:20Z
bmceventsubscriptions.metal3.io                                   2023-04-17T04:53:23Z
builds.config.openshift.io                                        2023-04-17T04:52:40Z
catalogsources.operators.coreos.com                               2023-04-17T04:53:22Z
cephblockpoolradosnamespaces.ceph.rook.io                         2023-04-17T05:25:22Z
cephblockpools.ceph.rook.io                                       2023-04-17T05:25:23Z
cephbucketnotifications.ceph.rook.io                              2023-04-17T05:25:23Z
cephbuckettopics.ceph.rook.io                                     2023-04-17T05:25:24Z
cephclients.ceph.rook.io                                          2023-04-17T05:25:24Z
cephclusters.ceph.rook.io                                         2023-04-17T05:25:23Z
cephfilesystemmirrors.ceph.rook.io                                2023-04-17T05:25:22Z
cephfilesystems.ceph.rook.io                                      2023-04-17T05:25:23Z
cephfilesystemsubvolumegroups.ceph.rook.io                        2023-04-17T05:25:24Z
cephnfses.ceph.rook.io                                            2023-04-17T05:25:22Z
cephobjectrealms.ceph.rook.io                                     2023-04-17T05:25:22Z
cephobjectstores.ceph.rook.io                                     2023-04-17T05:25:22Z
cephobjectstoreusers.ceph.rook.io                                 2023-04-17T05:25:22Z
cephobjectzonegroups.ceph.rook.io                                 2023-04-17T05:25:22Z
cephobjectzones.ceph.rook.io                                      2023-04-17T05:25:23Z
cephrbdmirrors.ceph.rook.io                                       2023-04-17T05:25:22Z
cloudcredentials.operator.openshift.io                            2023-04-17T04:53:04Z
cloudprivateipconfigs.cloud.network.openshift.io                  2023-04-17T04:55:04Z
clusterautoscalers.autoscaling.openshift.io                       2023-04-17T04:53:21Z
clustercsidrivers.operator.openshift.io                           2023-04-17T04:54:02Z
clusteroperators.config.openshift.io                              2023-04-17T04:52:28Z
clusterresourcequotas.quota.openshift.io                          2023-04-17T04:52:39Z
clusterserviceversions.operators.coreos.com                       2023-04-17T04:53:24Z
clusterurlmonitors.monitoring.openshift.io                        2023-04-17T05:21:20Z
clusterversions.config.openshift.io                               2023-04-17T04:52:28Z
configs.imageregistry.operator.openshift.io                       2023-04-17T04:53:19Z
configs.operator.openshift.io                                     2023-04-17T04:53:26Z
configs.samples.operator.openshift.io                             2023-04-17T04:53:17Z
consoleclidownloads.console.openshift.io                          2023-04-17T04:53:18Z
consoleexternalloglinks.console.openshift.io                      2023-04-17T04:53:18Z
consolelinks.console.openshift.io                                 2023-04-17T04:53:17Z
consolenotifications.console.openshift.io                         2023-04-17T04:53:17Z
consoleplugins.console.openshift.io                               2023-04-17T04:53:17Z
consolequickstarts.console.openshift.io                           2023-04-17T04:53:17Z
consoles.config.openshift.io                                      2023-04-17T04:52:40Z
consoles.operator.openshift.io                                    2023-04-17T04:53:18Z
consoleyamlsamples.console.openshift.io                           2023-04-17T04:53:17Z
containerruntimeconfigs.machineconfiguration.openshift.io         2023-04-17T04:53:42Z
controllerconfigs.machineconfiguration.openshift.io               2023-04-17T04:56:48Z
controlplanemachinesets.machine.openshift.io                      2023-04-17T04:53:19Z
credentialsrequests.cloudcredential.openshift.io                  2023-04-17T04:53:04Z
csisnapshotcontrollers.operator.openshift.io                      2023-04-17T04:53:21Z
customdomains.managed.openshift.io                                2023-04-17T05:21:50Z
deletebackuprequests.velero.io                                    2023-04-17T05:22:18Z
dnses.config.openshift.io                                         2023-04-17T04:52:41Z
dnses.operator.openshift.io                                       2023-04-17T04:53:26Z
dnsrecords.ingress.operator.openshift.io                          2023-04-17T04:53:22Z
downloadrequests.velero.io                                        2023-04-17T05:22:18Z
egressfirewalls.k8s.ovn.org                                       2023-04-17T04:55:11Z
egressips.k8s.ovn.org                                             2023-04-17T04:55:12Z
egressqoses.k8s.ovn.org                                           2023-04-17T04:55:12Z
egressrouters.network.operator.openshift.io                       2023-04-17T04:53:33Z
etcds.operator.openshift.io                                       2023-04-17T04:53:18Z
featuregates.config.openshift.io                                  2023-04-17T04:52:41Z
firmwareschemas.metal3.io                                         2023-04-17T04:53:26Z
hardwaredata.metal3.io                                            2023-04-17T04:53:27Z
helmchartrepositories.helm.openshift.io                           2023-04-17T04:53:18Z
hostfirmwaresettings.metal3.io                                    2023-04-17T04:53:30Z
imagecontentpolicies.config.openshift.io                          2023-04-17T04:52:42Z
imagecontentsourcepolicies.operator.openshift.io                  2023-04-17T04:52:42Z
imagepruners.imageregistry.operator.openshift.io                  2023-04-17T04:53:52Z
images.config.openshift.io                                        2023-04-17T04:52:42Z
infrastructures.config.openshift.io                               2023-04-17T04:52:43Z
ingresscontrollers.operator.openshift.io                          2023-04-17T04:53:07Z
ingresses.config.openshift.io                                     2023-04-17T04:52:43Z
insightsoperators.operator.openshift.io                           2023-04-17T05:03:57Z
installplans.operators.coreos.com                                 2023-04-17T04:53:26Z
ippools.whereabouts.cni.cncf.io                                   2023-04-17T04:55:05Z
kubeapiservers.operator.openshift.io                              2023-04-17T04:53:56Z
kubecontrollermanagers.operator.openshift.io                      2023-04-17T04:53:23Z
kubeletconfigs.machineconfiguration.openshift.io                  2023-04-17T04:53:43Z
kubeschedulers.operator.openshift.io                              2023-04-17T04:53:23Z
kubestorageversionmigrators.operator.openshift.io                 2023-04-17T04:53:17Z
machineautoscalers.autoscaling.openshift.io                       2023-04-17T04:53:23Z
machineconfigpools.machineconfiguration.openshift.io              2023-04-17T04:53:47Z
machineconfigs.machineconfiguration.openshift.io                  2023-04-17T04:53:46Z
machinehealthchecks.machine.openshift.io                          2023-04-17T04:54:01Z
machines.machine.openshift.io                                     2023-04-17T04:53:59Z
machinesets.machine.openshift.io                                  2023-04-17T04:54:01Z
managedfleetnotificationrecords.ocmagent.managed.openshift.io     2023-04-17T05:21:29Z
managedfleetnotifications.ocmagent.managed.openshift.io           2023-04-17T05:21:29Z
managedfusionofferings.misf.ibm.com                               2023-04-17T05:23:38Z
managednotifications.ocmagent.managed.openshift.io                2023-04-17T05:21:29Z
monitoringstacks.monitoring.rhobs                                 2023-04-17T05:22:27Z
mustgathers.managed.openshift.io                                  2023-04-17T05:21:13Z
network-attachment-definitions.k8s.cni.cncf.io                    2023-04-17T04:55:04Z
networks.config.openshift.io                                      2023-04-17T04:52:44Z
networks.operator.openshift.io                                    2023-04-17T04:53:23Z
nodes.config.openshift.io                                         2023-04-17T04:52:44Z
noobaas.noobaa.io                                                 2023-04-17T05:24:15Z
oauths.config.openshift.io                                        2023-04-17T04:52:44Z
objectbucketclaims.objectbucket.io                                2023-04-17T05:24:15Z
objectbuckets.objectbucket.io                                     2023-04-17T05:24:15Z
ocmagents.ocmagent.managed.openshift.io                           2023-04-17T05:21:29Z
ocsinitializations.ocs.openshift.io                               2023-04-17T05:24:15Z
olmconfigs.operators.coreos.com                                   2023-04-17T04:53:33Z
openshiftapiservers.operator.openshift.io                         2023-04-17T04:53:18Z
openshiftcontrollermanagers.operator.openshift.io                 2023-04-17T04:53:24Z
operatorconditions.operators.coreos.com                           2023-04-17T04:53:36Z
operatorgroups.operators.coreos.com                               2023-04-17T04:53:37Z
operatorhubs.config.openshift.io                                  2023-04-17T04:53:18Z
operatorpkis.network.operator.openshift.io                        2023-04-17T04:53:36Z
operators.operators.coreos.com                                    2023-04-17T04:53:40Z
overlappingrangeipreservations.whereabouts.cni.cncf.io            2023-04-17T04:55:05Z
performanceprofiles.performance.openshift.io                      2023-04-17T04:53:23Z
podmonitors.monitoring.coreos.com                                 2023-04-17T04:53:26Z
podmonitors.monitoring.rhobs                                      2023-04-17T05:22:30Z
podnetworkconnectivitychecks.controlplane.operator.openshift.io   2023-04-17T05:20:26Z
podvolumebackups.velero.io                                        2023-04-17T05:22:18Z
podvolumerestores.velero.io                                       2023-04-17T05:22:18Z
preprovisioningimages.metal3.io                                   2023-04-17T04:53:32Z
probes.monitoring.coreos.com                                      2023-04-17T04:53:29Z
probes.monitoring.rhobs                                           2023-04-17T05:22:30Z
profiles.tuned.openshift.io                                       2023-04-17T04:53:26Z
projecthelmchartrepositories.helm.openshift.io                    2023-04-17T04:53:17Z
projects.config.openshift.io                                      2023-04-17T04:52:45Z
prometheuses.monitoring.coreos.com                                2023-04-17T04:53:30Z
prometheuses.monitoring.rhobs                                     2023-04-17T05:22:30Z
prometheusrules.monitoring.coreos.com                             2023-04-17T04:53:33Z
prometheusrules.monitoring.rhobs                                  2023-04-17T05:22:30Z
provisionings.metal3.io                                           2023-04-17T04:53:39Z
proxies.config.openshift.io                                       2023-04-17T04:52:38Z
rangeallocations.security.internal.openshift.io                   2023-04-17T04:52:39Z
resticrepositories.velero.io                                      2023-04-17T05:22:18Z
restores.velero.io                                                2023-04-17T05:22:18Z
rolebindingrestrictions.authorization.openshift.io                2023-04-17T04:52:38Z
routemonitors.monitoring.openshift.io                             2023-04-17T05:21:20Z
schedulers.config.openshift.io                                    2023-04-17T04:52:45Z
schedules.velero.io                                               2023-04-17T05:22:18Z
securitycontextconstraints.security.openshift.io                  2023-04-17T04:52:39Z
serverstatusrequests.velero.io                                    2023-04-17T05:22:18Z
servicecas.operator.openshift.io                                  2023-04-17T04:53:26Z
servicemonitors.monitoring.coreos.com                             2023-04-17T04:53:36Z
servicemonitors.monitoring.rhobs                                  2023-04-17T05:22:31Z
splunkforwarders.splunkforwarder.managed.openshift.io             2023-04-17T05:21:47Z
storageclassclaims.ocs.openshift.io                               2023-04-17T05:25:22Z
storageclusters.ocs.openshift.io                                  2023-04-17T05:24:14Z
storageconsumers.ocs.openshift.io                                 2023-04-17T05:25:22Z
storages.operator.openshift.io                                    2023-04-17T04:54:02Z
storagestates.migration.k8s.io                                    2023-04-17T04:53:24Z
storageversionmigrations.migration.k8s.io                         2023-04-17T04:53:21Z
subjectpermissions.managed.openshift.io                           2023-04-17T05:22:34Z
subscriptions.operators.coreos.com                                2023-04-17T04:53:56Z
thanosqueriers.monitoring.rhobs                                   2023-04-17T05:22:31Z
thanosrulers.monitoring.coreos.com                                2023-04-17T04:53:37Z
thanosrulers.monitoring.rhobs                                     2023-04-17T05:22:31Z
tuneds.tuned.openshift.io                                         2023-04-17T04:53:29Z
upgradeconfigs.upgrade.managed.openshift.io                       2023-04-17T05:22:19Z
veleroinstalls.managed.openshift.io                               2023-04-17T05:21:58Z
volumesnapshotclasses.snapshot.storage.k8s.io                     2023-04-17T04:57:01Z
volumesnapshotcontents.snapshot.storage.k8s.io                    2023-04-17T04:57:01Z
volumesnapshotlocations.velero.io                                 2023-04-17T05:22:18Z
volumesnapshots.snapshot.storage.k8s.io                           2023-04-17T04:57:01Z

Comment 21 Dhruv Bindra 2023-04-17 08:45:26 UTC

@tnielsen These are the CRDs that are not on the cluster when we install ocs operator but rook tries to watch the resource.
objectbuckets.objectbucket.io
objectbucketclaims.objectbucket.io

Comment 22 Travis Nielsen 2023-04-20 19:14:51 UTC

Thanks for confirming the CRDs on the system

Comment 23 Travis Nielsen 2023-04-24 16:43:50 UTC

It's merged upstream, Santosh please create the backport PR for downstream.

Comment 24 Travis Nielsen 2023-05-02 15:24:03 UTC

Merged a week ago, should be in the 4.13 build now

Comment 27 suchita 2023-06-19 08:20:40 UTC

Verified on OCP 4.13.2 , managed Fusion service setup, Provider and consumer with ocs operator OCS4.13.0-219. All pods up and running. 
New fass does not use openshift-storage namespace. instead, it used fusion-storage namespace. 

================================
$ oc get clusterversion
NAME      VERSION   AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.13.2    True        False         64m     Cluster version is 4.13.2
[jenkins@odf-ms-stage privateLink]$ oc get pods
NAME                                                              READY   STATUS      RESTARTS   AGE
d64ffb5a5747fd5d820a289a4c6c9fd801993e260539ac2c7f492e6aa42pj26   0/1     Completed   0          21m
managed-fusion-offering-catalog-x46z7                             1/1     Running     0          22m
ocs-metrics-exporter-7c7b7684f6-ccjfd                             1/1     Running     0          20m
ocs-operator-67c48d46b-vhcgc                                      1/1     Running     0          20m
ocs-provider-server-7c657594f8-xbcjn                              1/1     Running     0          19m
rook-ceph-crashcollector-48ba7b0f1ec398c93eabd05009fac8ee-snttd   1/1     Running     0          12m
rook-ceph-crashcollector-523b75183124fa2b333ac1c92b26ad47-49576   1/1     Running     0          11m
rook-ceph-crashcollector-de38c69a964c27d4abf986f19e56bda0-hbdrl   1/1     Running     0          12m
rook-ceph-exporter-ip-10-0-13-179.us-east-2.compute.internwgdbc   1/1     Running     0          11m
rook-ceph-exporter-ip-10-0-18-142.us-east-2.compute.internlv82v   1/1     Running     0          12m
rook-ceph-exporter-ip-10-0-22-250.us-east-2.compute.intern7v28v   1/1     Running     0          12m
rook-ceph-mds-ocs-storagecluster-cephfilesystem-a-fd468746bsxrz   2/2     Running     0          10m
rook-ceph-mds-ocs-storagecluster-cephfilesystem-b-5d489d484m4pp   2/2     Running     0          10m
rook-ceph-mgr-a-7fc45974cf-lmk7n                                  2/2     Running     0          12m
rook-ceph-mon-a-6cd6b44f7c-rswfz                                  2/2     Running     0          19m
rook-ceph-mon-b-7fb8cb76fb-4285m                                  2/2     Running     0          17m
rook-ceph-mon-c-5c59b799f7-8sm7c                                  2/2     Running     0          17m
rook-ceph-operator-76f766d76f-bdmtc                               1/1     Running     0          20m
rook-ceph-osd-0-85dc56b886-snlc6                                  2/2     Running     0          11m
rook-ceph-osd-1-768499d845-42nhv                                  2/2     Running     0          11m
rook-ceph-osd-2-75595d8ffc-kpnrt                                  2/2     Running     0          11m
rook-ceph-osd-prepare-default-0-data-08ttj6-kchh7                 0/1     Completed   0          11m
rook-ceph-osd-prepare-default-1-data-0sv6bw-ghxlz                 0/1     Completed   0          11m
rook-ceph-osd-prepare-default-2-data-0l8rhw-n8znv                 0/1     Completed   0          11m
rook-ceph-tools-d8cfd99c4-vtgdl                                   1/1     Running     0          19m

$ oc get csv
NAME                                      DISPLAY                       VERSION               REPLACES                                  PHASE
managed-fusion-agent.v2.0.11              Managed Fusion Agent          2.0.11                                                          Succeeded
observability-operator.v0.0.22            Observability Operator        0.0.22                observability-operator.v0.0.21            Succeeded
ocs-operator.v4.13.0-219.snaptrim         OpenShift Container Storage   4.13.0-219.snaptrim                                             Succeeded
ose-prometheus-operator.4.10.0            Prometheus Operator           4.10.0                                                          Succeeded
route-monitor-operator.v0.1.500-6152b76   Route Monitor Operator        0.1.500-6152b76       route-monitor-operator.v0.1.498-e33e391   Succeeded


$ oc get csv ocs-operator.v4.13.0-219.snaptrim -o yaml | grep full
    full_version: 4.13.0-219
=====================================

Based on the above output. Moving this BZ to verified

Comment 29 errata-xmlrpc 2023-06-21 15:25:01 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenShift Data Foundation 4.13.0 enhancement and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:3742

Comment 30 Red Hat Bugzilla 2023-12-08 04:32:59 UTC

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days

Note You need to log in before you can comment on or make changes to this bug.