Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2109096

Summary:	label openshift.io/cluster-monitoring is dropped after upgrade.
Product:	OpenShift Container Platform	Reporter:	German Parente <gparente>
Component:	Storage	Assignee:	Jan Safranek <jsafrane>
Storage sub component:	Local Storage Operator	QA Contact:	Chao Yang <chaoyang>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	medium
Priority:	medium	CC:	jsafrane
Version:	4.10
Target Milestone:	---
Target Release:	4.10.z
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-09-21 14:07:05 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	2052071
Bug Blocks:

Description German Parente 2022-07-20 11:58:08 UTC

Description of problem:

Customer has alert PrometheusOperatorRejectedResources appearing after upgrade.

After checking the operator logs it seems user workload prometheus is scrapping the service monitors:

openshift-local-storage/local-storage-discovery-metrics
openshift-local-storage/local-storage-diskmaker-metrics

and failing. And user workload monitoring is scrapping this namespace because the namespace is missing label

openshift.io/cluster-monitoring: "true"

But it cannot be set because the namespaces labels are managed by the local storage operator.


Version-Release number of selected component (if applicable): 4.10

Comment 1 Jan Safranek 2022-09-05 19:39:11 UTC

> And user workload monitoring is scrapping this namespace because the namespace is missing label
> openshift.io/cluster-monitoring: "true"
> But it cannot be set because the namespaces labels are managed by the local storage operator.

I found it weird that something removes label "openshift.io/cluster-monitoring" from namespace "openshift-local-storage" - if it was, Prometheus would not scrape the namespace at all and there would be no alert. I added the label via the following command and nothing removes it during upgrade:

$ oc patch namespace openshift-local-storage --type='json' -p='[{"op": "add", "path": "/metadata/labels", "value": {"openshift.io/cluster-monitoring":"true"}}]'


This bug looks like a dup of https://bugzilla.redhat.com/show_bug.cgi?id=2052071, I'll fix it in 4.10 soon.

Comment 2 Jan Safranek 2022-09-05 19:42:17 UTC

I'm reusing this BZ to track 4.10.z backport

Comment 4 Chao Yang 2022-09-14 06:46:01 UTC

Verification is passed when upgrade from 4.8 to 4.10 local-storage-operator.4.10.0-202209080237

Comment 7 errata-xmlrpc 2022-09-21 14:07:05 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.10.33 extras update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:6533