Bug 2050707

Summary:	up test for prometheus pod look to far in the past
Product:	OpenShift Container Platform	Reporter:	Jan Fajerski <jfajersk>
Component:	Monitoring	Assignee:	Jan Fajerski <jfajersk>
Status:	CLOSED ERRATA	QA Contact:	Junqi Zhao <juzhao>
Severity:	low	Docs Contact:
Priority:	unspecified
Version:	4.11	CC:	amuller, anpicker, hongyli
Target Milestone:	---
Target Release:	4.11.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-08-10 10:47:24 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Jan Fajerski 2022-02-04 13:50:25 UTC

Description of problem:
The up test for the prometheus pod (https://github.com/openshift/cluster-monitoring-operator/blob/650c320463c15b3f5a64abcd02a066c7a9c15b4a/test/e2e/main_test.go#L87) counts the number of series returned. Due to prometheus staleness notion this will return samples for all series that have received samples in the last 5 minutes.
Due to improved availability during 4.10 development a 5 minute interval can yield 4 results instead of the expected two. Since the test only polls this condition for one minute it can time out waiting to see 2 pods.


How reproducible:
Run the test linked above.

Actual results:
Test fails with message "wait for prometheus-k8s: expected 2 Prometheus instances but got: 4: timed out waiting for the condition"

Expected results:
Test passes

Additional info:
To fix this, use last_over_time aggregation to reduce the lookback period.

Comment 3 Junqi Zhao 2022-02-15 04:30:12 UTC

LGTM, Test passes

Comment 8 errata-xmlrpc 2022-08-10 10:47:24 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069