Bug 2055729 - NodePerfCheck fires and stays active on momentary high latency
Summary: NodePerfCheck fires and stays active on momentary high latency
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 4.10
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.11.0
Assignee: rvanderp
QA Contact: Penghao Wang
Depends On:
Blocks: 2055895
TreeView+ depends on / blocked
Reported: 2022-02-17 15:24 UTC by rvanderp
Modified: 2022-08-10 10:50 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Last Closed: 2022-08-10 10:50:22 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift vsphere-problem-detector pull 75 0 None open Bug 2055729: change the alert criteria to be based on average latency 2022-02-17 18:56:46 UTC
Red Hat Product Errata RHSA-2022:5069 0 None None None 2022-08-10 10:50:35 UTC

Description rvanderp 2022-02-17 15:24:27 UTC
Description of problem:

vSphere upgrade jobs are failing due to the NodePerfCheck alert firing.  The actual duration of the high latency is unclear.  Additionally, if the latency is momentary(i.e. high latency is encountered for 1 sample) the alert will remain active until the high latency metric rotates out of the history.

Version-Release number of selected component (if applicable):

How reproducible:
Consistently in 4.10 vSphere CI jobs

Steps to Reproduce:
1. Monitor the job periodic-ci-openshift-release-master-ci-4.10-upgrade-from-stable-4.9-e2e-vsphere-upgrade

Actual results:
Alert is firing and remains active for momentary high latency.

See: https://testgrid.k8s.io/redhat-openshift-ocp-release-4.10-informing#periodic-ci-openshift-release-master-ci-4.10-upgrade-from-stable-4.9-e2e-vsphere-upgrade

Expected results:
Alert should not fire and remain active for a momentary high latency event.

Master Log:

Node Log (of failed PODs):

PV Dump:

PVC Dump:

StorageClass Dump (if StorageClass used by PV/PVC):

Additional info:

Comment 6 errata-xmlrpc 2022-08-10 10:50:22 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.