Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: The documented way to enable heketi metrics does not work with OCP 3.11, since in OCP 3.11 Prometheus works with ServiceMonitors and not with ConfigMaps as in OCP 3.10. Version-Release number of selected component (if applicable): (OCS 3.11 & OCS 3.10) + OCP 3.11 How reproducible: Every time Steps to Reproduce: 1. # oc get cm prometheus -n openshift-monitoring 2. 3. Actual results: No resources found. Error from server (NotFound): configmaps "prometheus" not found Expected results: Description of Prometheus CM Additional info: According to Christian Heidenreich, the proper way to enable additional metrics (including heketi's) is to deploy an additional Prometheus.
Hi Patric, unless the metrics endpoint Heketi provides is failing to work correctly I don't really see this as a heketi bug per-se. Heketi itself doesn't control how Prometheus finds the endpoint. Can we find a more approprite component for this issue, like "CNS-deployment" perhaps or maybe a doc component?
Hi John, thanks, you're right. CNS-deployment is better.
Michael, who's looking into this? Metrics (and especially alerts we can have on top of them) are critical. It's unclear to me how QE tested it.
(In reply to Yaniv Kaul from comment #4) > Michael, who's looking into this? Further below. > Metrics (and especially alerts we can have on top of them) are critical. Agreed. > It's unclear to me how QE tested it. Same here. Here's the doc: https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/3.11/html-single/operations_guide/#enable_vol_metrics Heketi as such just provides a set of metrics when the `/metrics` url is queried. From reading the description of this BZ, it seems like this is a regression in OpenShift, i.e. the way prometheus is set up has changed in openshift 3.11, and hence our instructions do not work any more. So I think we will adapt the doc steps, and this will turn into a doc BZ. Steps: 1. Needinfo on Karthick / Kasturi to tell us whether this part has been tested for 3.11.0/3.11.1/3.11.2, and if so, how was it successful. 2. Needinfo on Humble /reassigning to figure out the correct steps for OCP 3.11 (or tell us who can take care of this).
(In reply to Michael Adam from comment #5) > (In reply to Yaniv Kaul from comment #4) > > Michael, who's looking into this? > > Further below. > > > Metrics (and especially alerts we can have on top of them) are critical. > > Agreed. > > > It's unclear to me how QE tested it. > > Same here. > > Here's the doc: > > https://access.redhat.com/documentation/en-us/ > red_hat_openshift_container_storage/3.11/html-single/operations_guide/ > #enable_vol_metrics > > Heketi as such just provides a set of metrics when the `/metrics` url is > queried. > > From reading the description of this BZ, it seems like this is a regression > in OpenShift, i.e. the way prometheus is set up has changed in openshift > 3.11, and hence our instructions do not work any more. So I think we will > adapt the doc steps, and this will turn into a doc BZ. > > Steps: > > 1. Needinfo on Karthick / Kasturi to tell us whether this part has been > tested for 3.11.0/3.11.1/3.11.2, and if so, how was it successful. We have it tested during 3.11 and 3.11.1 and have raised a doc bug to get this corrected. Below is the doc bug for the same which is getting fixed for 3.11.2 https://bugzilla.redhat.com/show_bug.cgi?id=1661218 > 2. Needinfo on Humble /reassigning to figure out the correct steps for OCP > 3.11 (or tell us who can take care of this). clearing the needinfo on me and krk since i have provided the requested details.
setting back the need info on humble as this was cleared with the above comment.
Thanks Kasturi, So we should just close this one as a duplicate of existing the doc bug and collaborate on that one for the solution. Patric, agreed?
(In reply to Michael Adam from comment #8) > Thanks Kasturi, > > So we should just close this one as a duplicate of existing the doc bug and > collaborate on that one for the solution. > > Patric, agreed? Hi Michael, the problem with the suggested doc approach I see is that it creates an additional ServiceMonitor in the openshift-monitoring namespace. OpenShift docs explicitly state this as an unsupported operation (https://access.redhat.com/documentation/en-us/openshift_container_platform/3.11/html-single/configuring_clusters/#persistent-storage): Explicitly unsupported cases include: Creating additional ServiceMonitor objects in the openshift-monitoring namespace, thereby extending the targets the cluster monitoring Prometheus instance scrapes. This can cause collisions and load differences that cannot be accounted for, therefore the Prometheus setup can be unstable. We'd need a properly supported solution.
Status?