Bug 2055895 - NodePerfCheck fires and stays active on momentary high latency
Summary: NodePerfCheck fires and stays active on momentary high latency
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 4.10
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.10.0
Assignee: rvanderp
QA Contact: Penghao Wang
Depends On: 2055729
TreeView+ depends on / blocked
Reported: 2022-02-17 21:22 UTC by OpenShift BugZilla Robot
Modified: 2022-04-08 05:04 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Last Closed: 2022-04-08 05:04:18 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift vsphere-problem-detector pull 76 0 None open [release-4.10] Bug 2055895: change NodePerfCheck criteria to be based on average latency 2022-02-17 21:22:53 UTC
Red Hat Product Errata RHSA-2022:1162 0 None None None 2022-04-08 05:04:41 UTC

Description OpenShift BugZilla Robot 2022-02-17 21:22:41 UTC
+++ This bug was initially created as a clone of Bug #2055729 +++

Description of problem:

vSphere upgrade jobs are failing due to the NodePerfCheck alert firing.  The actual duration of the high latency is unclear.  Additionally, if the latency is momentary(i.e. high latency is encountered for 1 sample) the alert will remain active until the high latency metric rotates out of the history.

Version-Release number of selected component (if applicable):

How reproducible:
Consistently in 4.10 vSphere CI jobs

Steps to Reproduce:
1. Monitor the job periodic-ci-openshift-release-master-ci-4.10-upgrade-from-stable-4.9-e2e-vsphere-upgrade

Actual results:
Alert is firing and remains active for momentary high latency.

See: https://testgrid.k8s.io/redhat-openshift-ocp-release-4.10-informing#periodic-ci-openshift-release-master-ci-4.10-upgrade-from-stable-4.9-e2e-vsphere-upgrade

Expected results:
Alert should not fire and remain active for a momentary high latency event.

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:

Comment 6 errata-xmlrpc 2022-04-08 05:04:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.10.8 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.