Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1849396

Summary: oVirt e2e: failing test "[k8s.io] [sig-node] Events [Top Level] [k8s.io] [sig-node] Events should be sent by kubelets and the scheduler about pods scheduling and running [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s] "
Product: OpenShift Container Platform Reporter: Gal Zaidman <gzaidman>
Component: NodeAssignee: Gal Zaidman <gzaidman>
Status: CLOSED DUPLICATE QA Contact: Sunil Choudhary <schoudha>
Severity: high Docs Contact:
Priority: high    
Version: 4.4CC: aos-bugs, jokerman
Target Milestone: ---   
Target Release: 4.6.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-06-21 14:17:24 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Gal Zaidman 2020-06-21 11:23:22 UTC 
Description of problem:

Recently the test case:

"[k8s.io] [sig-node] Events [Top Level] [k8s.io] [sig-node] Events should be sent by kubelets and the scheduler about pods scheduling and running [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s]"

As been failing oVirt conformance jobs across the board, can be seen on:
https://ci-search-ci-search-next.svc.ci.openshift.org/?search=Events+should+be+sent+by+kubelets+and+the+scheduler+about+pods+scheduling+and+running&maxAge=168h&context=2&type=junit

The test case always fails after "checking for kubelet event about the pod" step.

https://github.com/kubernetes/kubernetes/blob/master/test/e2e/node/events.go#L116-L135

And fails with:
fail [k8s.io/kubernetes/test/e2e/node/events.go:115]: Unexpected error:
    <*errors.errorString | 0xc0002911f0>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
occurred


Last time we saw this test case the problem was with the amount of CPU on the test case pod, when we increased it the test case stopped appearing.
Now that we check the CPU we see a normal amount of load average, which leads us to suspect the problem is different.


How reproducible:
- run e2e conformance, in most cases, will see that test case fail.
Comment 1 Gal Zaidman 2020-06-21 11:33:30 UTC 
When I check the events of the test namespace I saw 2 events:

sh-4.2$ oc get events
LAST SEEN   TYPE     REASON           OBJECT                                                 MESSAGE
<unknown>   Normal   Scheduled        pod/send-events-70a2ee29-a18e-4c76-9249-b401637887de   Successfully assigned e2e-events-3759/send-events-70a2ee29-a18e-4c76-9249-b401637887de to ovirt10-cqjjn-worker-0-vtc8g
2m20s       Normal   AddedInterface   pod/send-events-70a2ee29-a18e-4c76-9249-b401637887de   Add eth0 [10.131.0.153/23]

sh-4.2$ oc get events -o yaml
apiVersion: v1
items:
- action: Binding
...
  involvedObject:
    apiVersion: v1
    kind: Pod
    name: send-events-70a2ee29-a18e-4c76-9249-b401637887de
    namespace: e2e-events-3759
    resourceVersion: "145028"
    uid: 0dc2eea5-94d8-4156-8a31-3769264b71eb
  kind: Event
  lastTimestamp: null
  message: Successfully assigned e2e-events-3759/send-events-70a2ee29-a18e-4c76-9249-b401637887de
    to ovirt10-cqjjn-worker-0-vtc8g
....
  reason: Scheduled
  reportingComponent: default-scheduler
  reportingInstance: default-scheduler-ovirt10-cqjjn-master-2
  source:
    component: default-scheduler
  type: Normal
- apiVersion: v1
...
  involvedObject:
    apiVersion: v1
    kind: Pod
    name: send-events-70a2ee29-a18e-4c76-9249-b401637887de
    namespace: e2e-events-3759
    resourceVersion: "145037"
    uid: 0dc2eea5-94d8-4156-8a31-3769264b71eb
  kind: Event
  lastTimestamp: "2020-06-21T11:25:09Z"
  message: Add eth0 [10.131.0.153/23]
  metadata:
    creationTimestamp: "2020-06-21T11:25:09Z"
...
  reason: AddedInterface
  reportingComponent: ""
  reportingInstance: ""
  source:
    component: multus
  type: Normal
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
Comment 2 Gal Zaidman 2020-06-21 14:17:24 UTC 

*** This bug has been marked as a duplicate of bug 1848081 ***