Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2027539

Summary: [sig-node] Managed cluster should report ready nodes the entire duration of the test run [Late] [Suite:openshift/conformance/parallel] failing on Prometheus query
Product: OpenShift Container Platform Reporter: Aravindh Puthiyaparambil <aravindh>
Component: NodeAssignee: Swarup Ghosh <swghosh>
Node sub component: Kubelet QA Contact: Weinan Liu <weinliu>
Status: CLOSED ERRATA Docs Contact:
Severity: medium    
Priority: medium CC: aos-bugs, harpatil, nagrawal, sippy, weinliu, wking
Version: 4.7   
Target Milestone: ---   
Target Release: 4.7.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-01-19 13:30:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Aravindh Puthiyaparambil 2021-11-29 23:48:49 UTC 
This bug is preventing 4.7-ci payloads (https://openshift-release.apps.ci.l2s4.p1.openshiftapps.com/#4.7.0-0.ci) from being created

[sig-node] Managed cluster should report ready nodes the entire duration of the test run [Late] [Suite:openshift/conformance/parallel]

is failing frequently in CI, see:
https://sippy.ci.openshift.org/sippy-ng/tests/4.7/analysis?test=%5Bsig-node%5D%20Managed%20cluster%20should%20report%20ready%20nodes%20the%20entire%20duration%20of%20the%20test%20run%20%5BLate%5D%20%5BSuite%3Aopenshift%2Fconformance%2Fparallel%5D

Job URI: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-ci-4.7-e2e-aws-serial/1465429442523303936

Log:

STEP: Destroying namespace "e2e-test-managed-cluster-node-6rlpt" for this suite.
fail [github.com/openshift/origin/test/extended/machines/cluster.go:90]: Unexpected error:
    <errors.aggregate | len:1, cap:1>: [
        {
            s: "query failed: (min_over_time((max by (node) (kube_node_status_condition{condition=\"Ready\",status=\"true\"}) and (((max by (node) (kube_node_status_condition))) and (0*max by (node) (kube_node_status_condition offset 6m))))[47m3s:1s])) < 1: promQL query: (min_over_time((max by (node) (kube_node_status_condition{condition=\"Ready\",status=\"true\"}) and (((max by (node) (kube_node_status_condition))) and (0*max by (node) (kube_node_status_condition offset 6m))))[47m3s:1s])) < 1 had reported incorrect results:\n[{\"metric\":{\"node\":\"ip-10-0-168-126.ec2.internal\"},\"value\":[1638225395.628,\"0\"]}]",
        },
    ]
    query failed: (min_over_time((max by (node) (kube_node_status_condition{condition="Ready",status="true"}) and (((max by (node) (kube_node_status_condition))) and (0*max by (node) (kube_node_status_condition offset 6m))))[47m3s:1s])) < 1: promQL query: (min_over_time((max by (node) (kube_node_status_condition{condition="Ready",status="true"}) and (((max by (node) (kube_node_status_condition))) and (0*max by (node) (kube_node_status_condition offset 6m))))[47m3s:1s])) < 1 had reported incorrect results:
    [{"metric":{"node":"ip-10-0-168-126.ec2.internal"},"value":[1638225395.628,"0"]}]
occurred
Comment 5 Weinan Liu 2022-01-13 09:14:36 UTC 
Do not see such issues for recent 7days CI test
https://search.ci.openshift.org/?search=query+failed%3A+%28min_over_time%28%28max+by+%28node%29+%28kube_node_status_condition%7Bcondition%3D%5C%22Ready%5C%22%2Cstatus%3D%5C%22true%5C%22%7D%29&maxAge=168h&context=1&type=bug%2Bjunit&name=&excludeName=&maxMatches=5&maxBytes=20971520&groupBy=job
Comment 7 errata-xmlrpc 2022-01-19 13:30:41 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.7.41 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:0117