Bug 2026577

Summary:	Observability - Cluster number is changed in Top 50 panel each time when refreshing the Grafana UI
Product:	Red Hat Advanced Cluster Management for Kubernetes	Reporter:	cqu
Component:	Core Services / Observability	Assignee:	Chunlin Yang <chuyang>
Status:	CLOSED CURRENTRELEASE	QA Contact:	cqu
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	rhacm-2.4	CC:	cedric.girard, crizzo, ecai, ming, mlele, xiyin
Target Milestone:	---	Keywords:	DeliveryBlocker
Target Release:	rhacm-2.4.2	Flags:	cqu: qe_test_coverage- bot-tracker-sync: rhacm-2.4+
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-10-03 20:19:59 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description cqu 2021-11-25 07:56:12 UTC

Description of the problem:
There are 4 managed clusters, they should be always listed in each panel. But when the dashboard is refreshed automatically or manually, the cluster number is changed in Top 50 panel sometimes.

Release version: 2.4.1

Operator snapshot version:

OCP version: 4.8

Browser Info: Firefox and Safari on Mac

Steps to reproduce:
1. Deployed MCO CR and there are several managed clusters
2. Open Grafana Dashboard - Cluster Overview
3. Manually refresh dashboard and check cluster number in Top 50 panel

Comment 1 cqu 2021-11-25 08:00:08 UTC

This issue happened frequently at the beginning when the environment is set up, but when I ran automation on this environment and some cases make thanos-ruler restart, the cluster number is stable in Top 50 panel now. I will keep checking this issue, and raise it to a higher priority once it comes again.

Comment 2 cqu 2021-11-25 09:44:23 UTC

@ecai @xiyin FYI.

Comment 3 cqu 2021-11-26 14:39:59 UTC

When redeployed Observability, I encountered this issue again. I think this issue is high impact on 2.4.1, should be a stop-ship issue.

Comment 4 bot-tracker-sync 2021-12-08 17:06:39 UTC

G2Bsync 978975996 comment 
 morvencao Thu, 25 Nov 2021 08:57:04 UTC 
 G2Bsync The root cause is at upstream thanos ruler, we filed an [issue](https://github.com/thanos-io/thanos/issues/4900) in thanos community, waiting for response...

Comment 5 juhsu 2022-01-14 18:13:35 UTC

*** Bug 2028071 has been marked as a duplicate of this bug. ***

Comment 6 cqu 2022-02-10 08:46:16 UTC

I verified this issue has been fixed by 2.4.2 RC2, thanks.