Description of problem: Customer has alert PrometheusOperatorRejectedResources appearing after upgrade. After checking the operator logs it seems user workload prometheus is scrapping the service monitors: openshift-local-storage/local-storage-discovery-metrics openshift-local-storage/local-storage-diskmaker-metrics and failing. And user workload monitoring is scrapping this namespace because the namespace is missing label openshift.io/cluster-monitoring: "true" But it cannot be set because the namespaces labels are managed by the local storage operator. Version-Release number of selected component (if applicable): 4.10
> And user workload monitoring is scrapping this namespace because the namespace is missing label > openshift.io/cluster-monitoring: "true" > But it cannot be set because the namespaces labels are managed by the local storage operator. I found it weird that something removes label "openshift.io/cluster-monitoring" from namespace "openshift-local-storage" - if it was, Prometheus would not scrape the namespace at all and there would be no alert. I added the label via the following command and nothing removes it during upgrade: $ oc patch namespace openshift-local-storage --type='json' -p='[{"op": "add", "path": "/metadata/labels", "value": {"openshift.io/cluster-monitoring":"true"}}]' This bug looks like a dup of https://bugzilla.redhat.com/show_bug.cgi?id=2052071, I'll fix it in 4.10 soon.
I'm reusing this BZ to track 4.10.z backport
Verification is passed when upgrade from 4.8 to 4.10 local-storage-operator.4.10.0-202209080237
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.10.33 extras update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:6533