Bug 2055729

Summary:	NodePerfCheck fires and stays active on momentary high latency
Product:	OpenShift Container Platform	Reporter:	rvanderp
Component:	Storage	Assignee:	rvanderp
Storage sub component:	Operators	QA Contact:	Penghao Wang <pewang>
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	unspecified	CC:	sdodson, wking
Version:	4.10
Target Milestone:	---
Target Release:	4.11.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-08-10 10:50:22 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2055895

Description rvanderp 2022-02-17 15:24:27 UTC

Description of problem:

vSphere upgrade jobs are failing due to the NodePerfCheck alert firing.  The actual duration of the high latency is unclear.  Additionally, if the latency is momentary(i.e. high latency is encountered for 1 sample) the alert will remain active until the high latency metric rotates out of the history.

Version-Release number of selected component (if applicable):
4.10

How reproducible:
Consistently in 4.10 vSphere CI jobs

Steps to Reproduce:
1. Monitor the job periodic-ci-openshift-release-master-ci-4.10-upgrade-from-stable-4.9-e2e-vsphere-upgrade
2.
3.

Actual results:
Alert is firing and remains active for momentary high latency.

See: https://testgrid.k8s.io/redhat-openshift-ocp-release-4.10-informing#periodic-ci-openshift-release-master-ci-4.10-upgrade-from-stable-4.9-e2e-vsphere-upgrade

Expected results:
Alert should not fire and remain active for a momentary high latency event.

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:

Comment 6 errata-xmlrpc 2022-08-10 10:50:22 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069