+++ This bug was initially created as a clone of Bug #2188053 +++ Description of problem (please be detailed as possible and provide log snippests): ocs-metrics-exporter cannot list/watch StorageCluster, StorageClass, CephBlockPool and other resources it tries to look at openshift-storage for rook-ceph-mon secret ----------------------------From ocs-metrics-exporter log------------ W0419 14:31:47.825302 1 reflector.go:347] /remote-source/app/metrics/internal/collectors/registry.go:86: watch of *v1.CephBlockPool ended with: failed to initialize ceph: failed to get secret in namespace "openshift-storage": secrets "rook-ceph-mon" not found W0419 14:31:54.975735 1 reflector.go:424] /remote-source/app/metrics/internal/collectors/storage-cluster.go:42: failed to list *v1.StorageCluster: the server could not find the requested resource (get storageclusters.ocs.openshift.io) E0419 14:31:54.975765 1 reflector.go:140] /remote-source/app/metrics/internal/collectors/storage-cluster.go:42: Failed to watch *v1.StorageCluster: failed to list *v1.StorageCluster: the server could not find the requested resource (get storageclusters.ocs.openshift.io) I0419 14:32:16.082296 1 rbd-mirror.go:282] RBD mirror store resync started at 2023-04-19 14:32:16.082277933 +0000 UTC m=+32880.814374210 I0419 14:32:16.082368 1 rbd-mirror.go:307] RBD mirror store resync ended at 2023-04-19 14:32:16.082363603 +0000 UTC m=+32880.814459819 W0419 14:32:37.617111 1 reflector.go:424] /remote-source/app/metrics/internal/collectors/cluster-advance-feature-use.go:166: failed to list *v1.StorageClass: forbidden: User "system:serviceaccount:fusion-storage:ocs-metrics-exporter" cannot get path "/storageclasses" E0419 14:32:37.617142 1 reflector.go:140] /remote-source/app/metrics/internal/collectors/cluster-advance-feature-use.go:166: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: forbidden: User "system:serviceaccount:fusion-storage:ocs-metrics-exporter" cannot get path "/storageclasses" I0419 14:32:46.083431 1 rbd-mirror.go:282] RBD mirror store resync started at 2023-04-19 14:32:46.083420828 +0000 UTC m=+32910.815517141 I0419 14:32:46.083461 1 rbd-mirror.go:307] RBD mirror store resync ended at 2023-04-19 14:32:46.083457715 +0000 UTC m=+32910.815553931 W0419 14:32:49.307224 1 reflector.go:424] /remote-source/app/metrics/internal/collectors/storage-cluster.go:42: failed to list *v1.StorageCluster: the server could not find the requested resource (get storageclusters.ocs.openshift.io) E0419 14:32:49.307259 1 reflector.go:140] /remote-source/app/metrics/internal/collectors/storage-cluster.go:42: Failed to watch *v1.StorageCluster: failed to list *v1.StorageCluster: the server could not find the requested resource (get storageclusters.ocs.openshift.io) I0419 14:33:10.830451 1 ceph-blocklist.go:103] Blocklist store sync started 2023-04-19 14:33:10.830428943 +0000 UTC m=+32935.562525246 W0419 14:33:10.834664 1 reflector.go:347] /remote-source/app/metrics/internal/collectors/registry.go:86: watch of *v1.CephBlockPool ended with: failed to initialize ceph: failed to get secret in namespace "openshift-storage": secrets "rook-ceph-mon" not found W0419 14:33:11.755266 1 reflector.go:424] /remote-source/app/metrics/internal/collectors/cluster-advance-feature-use.go:166: failed to list *v1.StorageClass: forbidden: User "system:serviceaccount:fusion-storage:ocs-metrics-exporter" cannot get path "/storageclasses" E0419 14:33:11.755299 1 reflector.go:140] /remote-source/app/metrics/internal/collectors/cluster-advance-feature-use.go:166: Failed to watch *v1.StorageClass: failed to list *v1.StorageClass: forbidden: User "system:serviceaccount:fusion-storage:ocs-metrics-exporter" cannot get path "/storageclasses" --------------------------------------------------------------- Version of all relevant components (if applicable): $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.11.36 True False 10h $ oc get csv -n fusion-storage NAME DISPLAY VERSION REPLACES PHASE managed-fusion-agent.v2.0.11 Managed Fusion Agent 2.0.11 Succeeded observability-operator.v0.0.20 Observability Operator 0.0.20 observability-operator.v0.0.19 Succeeded ocs-operator.v4.13.0-164.stable OpenShift Container Storage 4.13.0-164.stable Succeeded ose-prometheus-operator.4.10.0 Prometheus Operator 4.10.0 Succeeded route-monitor-operator.v0.1.494-a973226 Route Monitor Operator 0.1.494-a973226 route-monitor-operator.v0.1.493-a866e7c Succeeded Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? yes Is there any workaround available to the best of your knowledge? no Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 1 Can this issue reproducible? 4/4 Can this issue reproduce from the UI? no If this is a regression, please provide more details to justify this: Steps to Reproduce: 1. Deployed the cluster with ocs operator 4.13.0-164 with managed fusion deployment steps 2. fusion-storage namespace used 3. Actual results: ocs-metrics-exporter cannot list/watch StorageCluster, StorageClass, CephBlockPool and other resources Expected results: ocs-metrics-exporter should able to list/watch StorageCluster, StorageClass, CephBlockPool and other resources Additional info: --- Additional comment from RHEL Program Management on 2023-04-19 15:33:59 UTC --- This bug having no release flag set previously, is now set with release flag 'odf‑4.13.0' to '?', and so is being proposed to be fixed at the ODF 4.13.0 release. Note that the 3 Acks (pm_ack, devel_ack, qa_ack), if any previously set while release flag was missing, have now been reset since the Acks are to be set against a release flag.
Removing 4.12.3 from IWB, as the issue is still under investigation (as per https://bugzilla.redhat.com/show_bug.cgi?id=2188053#c5)
We don't require PR#2036, 'Namespace agnostic metric exporter code', is not needed for release-4.12. The issue was created by a refactor change which is not backported to 4.12 branch (but release-4.13 require the changes). So I'm closed the PR
ocs-metrics-exporter pod logs are still full of errors that user "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list a resource: W0502 07:17:26.223240 1 reflector.go:324] /remote-source/app/metrics/internal/collectors/ceph-cluster.go:54: failed to list *v1.CephCluster: cephclusters.ceph.rook.io is forbidden: User "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list resource "cephclusters" in API group "ceph.rook.io" at the cluster scope E0502 07:17:26.223276 1 reflector.go:138] /remote-source/app/metrics/internal/collectors/ceph-cluster.go:54: Failed to watch *v1.CephCluster: failed to list *v1.CephCluster: cephclusters.ceph.rook.io is forbidden: User "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list resource "cephclusters" in API group "ceph.rook.io" at the cluster scope W0502 07:17:26.790539 1 reflector.go:324] /remote-source/app/metrics/internal/collectors/cluster-advance-feature-use.go:167: failed to list *v1.CephObjectStore: cephobjectstores.ceph.rook.io is forbidden: User "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list resource "cephobjectstores" in API group "ceph.rook.io" at the cluster scope E0502 07:17:26.790568 1 reflector.go:138] /remote-source/app/metrics/internal/collectors/cluster-advance-feature-use.go:167: Failed to watch *v1.CephObjectStore: failed to list *v1.CephObjectStore: cephobjectstores.ceph.rook.io is forbidden: User "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list resource "cephobjectstores" in API group "ceph.rook.io" at the cluster scope W0502 07:17:42.456513 1 reflector.go:324] /remote-source/app/metrics/internal/collectors/cluster-advance-feature-use.go:167: failed to list *v1.CephRBDMirror: cephrbdmirrors.ceph.rook.io is forbidden: User "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list resource "cephrbdmirrors" in API group "ceph.rook.io" at the cluster scope E0502 07:17:42.456535 1 reflector.go:138] /remote-source/app/metrics/internal/collectors/cluster-advance-feature-use.go:167: Failed to watch *v1.CephRBDMirror: failed to list *v1.CephRBDMirror: cephrbdmirrors.ceph.rook.io is forbidden: User "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list resource "cephrbdmirrors" in API group "ceph.rook.io" at the cluster scope W0502 07:17:51.119555 1 reflector.go:324] /remote-source/app/metrics/internal/collectors/cluster-advance-feature-use.go:167: failed to list *v1.CephCluster: cephclusters.ceph.rook.io is forbidden: User "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list resource "cephclusters" in API group "ceph.rook.io" at the cluster scope E0502 07:17:51.119581 1 reflector.go:138] /remote-source/app/metrics/internal/collectors/cluster-advance-feature-use.go:167: Failed to watch *v1.CephCluster: failed to list *v1.CephCluster: cephclusters.ceph.rook.io is forbidden: User "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list resource "cephclusters" in API group "ceph.rook.io" at the cluster scope W0502 07:17:53.276759 1 reflector.go:324] /remote-source/app/metrics/internal/collectors/registry.go:52: failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list resource "persistentvolumes" in API group "" at the cluster scope E0502 07:17:53.276782 1 reflector.go:138] /remote-source/app/metrics/internal/collectors/registry.go:52: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list resource "persistentvolumes" in API group "" at the cluster scope E0502 07:17:57.746379 1 reflector.go:138] /remote-source/app/metrics/internal/collectors/cluster-advance-feature-use.go:167: Failed to watch *v1.StorageClass: unknown (get storageclasses.storage.k8s.io) W0502 07:17:57.882468 1 reflector.go:324] /remote-source/app/metrics/internal/collectors/ceph-object-store.go:52: failed to list *v1.CephObjectStore: cephobjectstores.ceph.rook.io is forbidden: User "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list resource "cephobjectstores" in API group "ceph.rook.io" at the cluster scope E0502 07:17:57.882490 1 reflector.go:138] /remote-source/app/metrics/internal/collectors/ceph-object-store.go:52: Failed to watch *v1.CephObjectStore: failed to list *v1.CephObjectStore: cephobjectstores.ceph.rook.io is forbidden: User "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list resource "cephobjectstores" in API group "ceph.rook.io" at the cluster scope W0502 07:17:57.909313 1 reflector.go:324] /remote-source/app/metrics/internal/collectors/registry.go:64: failed to list *v1.CephBlockPool: cephblockpools.ceph.rook.io is forbidden: User "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list resource "cephblockpools" in API group "ceph.rook.io" at the cluster scope E0502 07:17:57.909331 1 reflector.go:138] /remote-source/app/metrics/internal/collectors/registry.go:64: Failed to watch *v1.CephBlockPool: failed to list *v1.CephBlockPool: cephblockpools.ceph.rook.io is forbidden: User "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list resource "cephblockpools" in API group "ceph.rook.io" at the cluster scope W0502 07:17:58.576788 1 reflector.go:324] /remote-source/app/metrics/internal/collectors/object-bucket.go:101: failed to list *v1.CephObjectStore: cephobjectstores.ceph.rook.io is forbidden: User "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list resource "cephobjectstores" in API group "ceph.rook.io" at the cluster scope E0502 07:17:58.576812 1 reflector.go:138] /remote-source/app/metrics/internal/collectors/object-bucket.go:101: Failed to watch *v1.CephObjectStore: failed to list *v1.CephObjectStore: cephobjectstores.ceph.rook.io is forbidden: User "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list resource "cephobjectstores" in API group "ceph.rook.io" at the cluster scope W0502 07:18:03.414074 1 reflector.go:324] /remote-source/app/metrics/internal/collectors/ceph-block-pool.go:61: failed to list *v1.CephBlockPool: cephblockpools.ceph.rook.io is forbidden: User "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list resource "cephblockpools" in API group "ceph.rook.io" at the cluster scope E0502 07:18:03.414096 1 reflector.go:138] /remote-source/app/metrics/internal/collectors/ceph-block-pool.go:61: Failed to watch *v1.CephBlockPool: failed to list *v1.CephBlockPool: cephblockpools.ceph.rook.io is forbidden: User "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list resource "cephblockpools" in API group "ceph.rook.io" at the cluster scope W0502 07:18:06.588442 1 reflector.go:324] /remote-source/app/metrics/internal/collectors/cluster-advance-feature-use.go:167: failed to list *v1.CephObjectStore: cephobjectstores.ceph.rook.io is forbidden: User "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list resource "cephobjectstores" in API group "ceph.rook.io" at the cluster scope E0502 07:18:06.588479 1 reflector.go:138] /remote-source/app/metrics/internal/collectors/cluster-advance-feature-use.go:167: Failed to watch *v1.CephObjectStore: failed to list *v1.CephObjectStore: cephobjectstores.ceph.rook.io is forbidden: User "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list resource "cephobjectstores" in API group "ceph.rook.io" at the cluster scope W0502 07:18:08.177867 1 reflector.go:324] /remote-source/app/metrics/internal/collectors/ceph-cluster.go:54: failed to list *v1.CephCluster: cephclusters.ceph.rook.io is forbidden: User "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list resource "cephclusters" in API group "ceph.rook.io" at the cluster scope E0502 07:18:08.177894 1 reflector.go:138] /remote-source/app/metrics/internal/collectors/ceph-cluster.go:54: Failed to watch *v1.CephCluster: failed to list *v1.CephCluster: cephclusters.ceph.rook.io is forbidden: User "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list resource "cephclusters" in API group "ceph.rook.io" at the cluster scope W0502 07:18:24.420782 1 reflector.go:324] /remote-source/app/metrics/internal/collectors/cluster-advance-feature-use.go:167: failed to list *v1.CephCluster: cephclusters.ceph.rook.io is forbidden: User "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list resource "cephclusters" in API group "ceph.rook.io" at the cluster scope E0502 07:18:24.420805 1 reflector.go:138] /remote-source/app/metrics/internal/collectors/cluster-advance-feature-use.go:167: Failed to watch *v1.CephCluster: failed to list *v1.CephCluster: cephclusters.ceph.rook.io is forbidden: User "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list resource "cephclusters" in API group "ceph.rook.io" at the cluster scope W0502 07:18:25.246001 1 reflector.go:324] /remote-source/app/metrics/internal/collectors/registry.go:52: failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list resource "persistentvolumes" in API group "" at the cluster scope E0502 07:18:25.246025 1 reflector.go:138] /remote-source/app/metrics/internal/collectors/registry.go:52: Failed to watch *v1.PersistentVolume: failed to list *v1.PersistentVolume: persistentvolumes is forbidden: User "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list resource "persistentvolumes" in API group "" at the cluster scope W0502 07:18:28.308082 1 reflector.go:324] /remote-source/app/metrics/internal/collectors/registry.go:64: failed to list *v1.CephBlockPool: cephblockpools.ceph.rook.io is forbidden: User "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list resource "cephblockpools" in API group "ceph.rook.io" at the cluster scope E0502 07:18:28.308108 1 reflector.go:138] /remote-source/app/metrics/internal/collectors/registry.go:64: Failed to watch *v1.CephBlockPool: failed to list *v1.CephBlockPool: cephblockpools.ceph.rook.io is forbidden: User "system:serviceaccount:openshift-storage:ocs-metrics-exporter" cannot list resource "cephblockpools" in API group "ceph.rook.io" at the cluster scope E0502 07:18:32.894142 1 reflector.go:138] /remote-source/app/metrics/internal/collectors/cluster-advance-feature-use.go:167: Failed to watch *v1.StorageClass: unknown (get storageclasses.storage.k8s.io) Resources listed in logs: CephCluster, CephObjectStore, CephRBDMirror, PersistentVolume, StorageClass and CephBlockPool. --> ASSIGNED Tested with: odf-operator v4.12.3-12 ocp 4.12.14
It seems like the clusterrole applied to ocs-metrics-exporter is not right. To find the which clusterrole is being applied (adding this for my own ref) Command: oc get clusterrolebindings -o custom-columns='KIND:kind,NAMESPACE:metadata.namespace,NAME:metadata.name,SERVICE_ACCOUNTS:subjects[?(@.kind=="ServiceAccount")].name' |grep "ocs-metrics-exporter" Sample output: ``` ClusterRoleBinding <none> ocs-operator.v4.12.3-rhodf-ffbbd588c ocs-metrics-exporter ``` Use the above clusterrolebinding-name to find the clusterrole used by ocs-metrics-exporter Command: oc get clusterrolebindings ocs-operator.v4.12.3-rhodf-ffbbd588c -o yaml Output will have a `roleRef:` attribute. Providing a sample output: ``` roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: ocs-operator.v4.12.3-rhodf-ffbbd588c ``` Get the clusterrole referred by the above binding, Command: oc get clusterroles ocs-operator.v4.12.3-rhodf-ffbbd588c -o yaml This is the actual output from the actual 4.12.3 cluster (where the error is occurring) Output: ``` apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: creationTimestamp: "2023-05-04T01:40:44Z" labels: olm.owner: ocs-operator.v4.12.3-rhodf olm.owner.kind: ClusterServiceVersion olm.owner.namespace: openshift-storage operators.coreos.com/ocs-operator.openshift-storage: "" name: ocs-operator.v4.12.3-rhodf-ffbbd588c resourceVersion: "202910" uid: 459947e4-236f-4fbc-b143-4aec987427f9 rules: - apiGroups: - monitoring.coreos.com resources: - '*' verbs: - '*' ``` There are not much permissions provided here Further checking...
On an ODF 4.13 cluster, we could see the following TWO clusterrolebindings applied, oc get clusterrolebindings -o custom-columns='KIND:kind,NAMESPACE:metadata.namespace,NAME:metadata.name,SERVICE_ACCOUNTS:subjects[?(@.kind=="ServiceAccount")].name' |rg "ocs-metrics-exporter" ClusterRoleBinding <none> ocs-operator.v4.13.0-181.stable-66c8494d79 ocs-metrics-exporter ClusterRoleBinding <none> ocs-operator.v4.13.0-181.stable-6f9d89c9d6 ocs-metrics-exporter If we output the ClusterRole associated with the second binding, ocs-operator.v4.13.0-181.stable-6f9d89c9d6, it has all the needed permissions ``` - apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: creationTimestamp: "2023-05-04T05:36:06Z" labels: olm.owner: ocs-operator.v4.13.0-181.stable olm.owner.kind: ClusterServiceVersion olm.owner.namespace: openshift-storage operators.coreos.com/ocs-operator.openshift-storage: "" name: ocs-operator.v4.13.0-181.stable-6f9d89c9d6 resourceVersion: "57713" uid: 1eb9d890-a9f0-4361-ae2b-03ff37fb5000 rules: - apiGroups: - ceph.rook.io resources: - cephobjectstores - cephblockpools - cephclusters - cephrbdmirrors verbs: - get - list - watch - apiGroups: - quota.openshift.io resources: - clusterresourcequotas verbs: - get - list - watch - apiGroups: - objectbucket.io resources: - objectbuckets verbs: - get - list - apiGroups: - "" resources: - configmaps - secrets verbs: - get - list - apiGroups: - "" resources: - persistentvolumes - persistentvolumeclaims - pods verbs: - get - list - watch - apiGroups: - storage.k8s.io resources: - storageclasses verbs: - get - list - watch - apiGroups: - ocs.openshift.io resources: - storageconsumers - storageclusters verbs: - get - list - watch kind: List metadata: resourceVersion: "" ``` _____________ In 4.12.3 we are missing this second ClusterRoleBinding...
Umanga has found that in 4.12.3 build is missing the required RBAC. Then this may not be a bug in the code base. As all the necessary changes are there in 4.12 release branch, we may just require a rebuild.
Additional Information from Cluster: Now on an ODF 4.12 cluster, we could see the following TWO cluster role bindings applied, ----------------------------------------------------------------------- $ oc get clusterrolebindings -o custom-columns='KIND:kind,NAMESPACE:metadata.namespace,NAME:metadata.name,SERVICE_ACCOUNTS:subjects[?(@.kind=="ServiceAccount")].name' | grep "ocs-metrics-exporter" ClusterRoleBinding <none> ocs-operator.v4.12.3-rhodf-6c598bf5b ocs-metrics-exporter ClusterRoleBinding <none> ocs-operator.v4.12.3-rhodf-7f46c8fb95 ----------------------------------------------------------------------- Permissions are found in one of the clusterRoleBinding $ oc get clusterrole ocs-operator.v4.12.3-rhodf-6c598bf5b -o yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: creationTimestamp: "2023-05-09T06:58:13Z" labels: olm.owner: ocs-operator.v4.12.3-rhodf olm.owner.kind: ClusterServiceVersion olm.owner.namespace: fusion-storage operators.coreos.com/ocs-operator.fusion-storage: "" name: ocs-operator.v4.12.3-rhodf-6c598bf5b resourceVersion: "93116" uid: 2d9f7a3c-0417-4841-9e39-5a5996c3e216 rules: - apiGroups: - ceph.rook.io resources: - cephobjectstores - cephblockpools - cephclusters - cephrbdmirrors verbs: - get - list - watch - apiGroups: - quota.openshift.io resources: - clusterresourcequotas verbs: - get - list - watch - apiGroups: - objectbucket.io resources: - objectbuckets verbs: - get - list - apiGroups: - "" resources: - configmaps - secrets verbs: - get - list - apiGroups: - "" resources: - persistentvolumes verbs: - get - list - watch - apiGroups: - storage.k8s.io resources: - storageclasses verbs: - get - list - watch --------------------------------- $ oc get clusterrole ocs-operator.v4.12.3-rhodf-7f46c8fb95 -o yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: creationTimestamp: "2023-05-09T06:58:13Z" labels: olm.owner: ocs-operator.v4.12.3-rhodf olm.owner.kind: ClusterServiceVersion olm.owner.namespace: fusion-storage operators.coreos.com/ocs-operator.fusion-storage: "" name: ocs-operator.v4.12.3-rhodf-7f46c8fb95 resourceVersion: "93117" uid: 8090c531-eb20-407a-8cc4-1ba4c0f7cf70 rules: - apiGroups: - monitoring.coreos.com resources: - '*' verbs: - '*' ---------------------------------------------------- $ oc get pods | grep ocs-metrics-exporter ocs-metrics-exporter-58fdb558d-s4dz9 1/1 Running 1 (5h31m ago) 6h14m $oc logs ocs-metrics-exporter-58fdb558d-s4dz9 I0509 07:42:15.619727 1 main.go:29] using options: &{Apiserver: KubeconfigPath: Host:0.0.0.0 Port:8080 ExporterHost:0.0.0.0 ExporterPort:8081 Help:false AllowedNamespaces:[fusion-storage] flags:0xc00015b100 StopCh:<nil> Kubeconfig:<nil>} W0509 07:42:15.619831 1 client_config.go:617] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. I0509 07:42:15.720012 1 main.go:70] Running metrics server on 0.0.0.0:8080 I0509 07:42:15.720035 1 main.go:71] Running telemetry server on 0.0.0.0:8081 I0509 07:42:16.220121 1 rbd-mirror.go:194] skipping rbd mirror status update for pool fusion-storage/cephblockpool-storageconsumer-281b0c3b-7b96-45a3-af7b-d7f7b11be0c2-70b813f6 because mirroring is disabled I0509 07:42:16.320171 1 pv.go:55] Skipping non Ceph CSI RBD volume pvc-dad6a948-b654-4ca1-a36d-c45bf5af6173 I0509 07:42:16.320196 1 pv.go:55] Skipping non Ceph CSI RBD volume pvc-08f61558-9611-4a06-a5c1-cccd90a0c28d I0509 07:42:16.320202 1 pv.go:55] Skipping non Ceph CSI RBD volume pvc-287244d4-dd0e-4795-8dbb-32744dfb6ad1 I0509 07:42:16.320206 1 pv.go:55] Skipping non Ceph CSI RBD volume pvc-b621cca7-8343-4501-8c02-91cdc4dee580 I0509 07:42:16.320211 1 pv.go:55] Skipping non Ceph CSI RBD volume pvc-6f67665b-207e-4e8e-aeac-b73433ed5c92 I0509 07:42:16.320216 1 pv.go:55] Skipping non Ceph CSI RBD volume pvc-3ab9b0a3-6b48-4ab1-bf46-826667578672 I0509 07:42:16.320220 1 pv.go:55] Skipping non Ceph CSI RBD volume pvc-dc0c2e38-685d-4b7b-ae71-ed6ff813f99d I0509 07:42:16.320225 1 pv.go:55] Skipping non Ceph CSI RBD volume pvc-60423614-06f9-4690-834f-7c41b7a508db I0509 07:42:16.320233 1 pv.go:55] Skipping non Ceph CSI RBD volume pvc-c853799c-a318-4675-a0db-ef76d34cc519 I0509 07:42:16.320237 1 pv.go:55] Skipping non Ceph CSI RBD volume pvc-8d634b54-ed58-499e-aac1-21652dc683ca I0509 07:42:20.882458 1 rbd-mirror.go:194] skipping rbd mirror status update for pool fusion-storage/cephblockpool-storageconsumer-281b0c3b-7b96-45a3-af7b-d7f7b11be0c2-70b813f6 because mirroring is disabled I0509 07:42:46.220750 1 rbd-mirror.go:273] RBD mirror store resync started at 2023-05-09 07:42:46.220740825 +0000 UTC m=+31.300752419 I0509 07:42:46.220804 1 rbd-mirror.go:298] RBD mirror store resync ended at 2023-05-09 07:42:46.220800947 +0000 UTC m=+31.300812533 E0509 07:43:12.200194 1 ceph-block-pool.go:137] Invalid image health for pool cephblockpool-storageconsumer-281b0c3b-7b96-45a3-af7b-d7f7b11be0c2-70b813f6. Must be OK, UNKNOWN, WARNING or ERROR ... ---------------------------------------------------------------------
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: Red Hat OpenShift Data Foundation 4.12.3 Security and Bug fix update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:3265