Bug 2055895

Summary: NodePerfCheck fires and stays active on momentary high latency
Product: OpenShift Container Platform Reporter: OpenShift BugZilla Robot <openshift-bugzilla-robot>
Component: StorageAssignee: rvanderp
Storage sub component: Operators QA Contact: Penghao Wang <pewang>
Status: CLOSED ERRATA Docs Contact:
Severity: high    
Priority: high CC: aos-bugs, sdodson, wking
Version: 4.10   
Target Milestone: ---   
Target Release: 4.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-04-08 05:04:18 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 2055729    
Bug Blocks:    

Description OpenShift BugZilla Robot 2022-02-17 21:22:41 UTC
+++ This bug was initially created as a clone of Bug #2055729 +++

Description of problem:

vSphere upgrade jobs are failing due to the NodePerfCheck alert firing.  The actual duration of the high latency is unclear.  Additionally, if the latency is momentary(i.e. high latency is encountered for 1 sample) the alert will remain active until the high latency metric rotates out of the history.

Version-Release number of selected component (if applicable):
4.10

How reproducible:
Consistently in 4.10 vSphere CI jobs

Steps to Reproduce:
1. Monitor the job periodic-ci-openshift-release-master-ci-4.10-upgrade-from-stable-4.9-e2e-vsphere-upgrade
2.
3.

Actual results:
Alert is firing and remains active for momentary high latency.

See: https://testgrid.k8s.io/redhat-openshift-ocp-release-4.10-informing#periodic-ci-openshift-release-master-ci-4.10-upgrade-from-stable-4.9-e2e-vsphere-upgrade

Expected results:
Alert should not fire and remain active for a momentary high latency event.

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:

Comment 6 errata-xmlrpc 2022-04-08 05:04:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.10.8 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:1162