Bug 1765301 - [k8s.io] [sig-node] Pods Extended [k8s.io] Delete Grace Period should be submitted and removed [Conformance]
Summary: [k8s.io] [sig-node] Pods Extended [k8s.io] Delete Grace Period should be subm...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 4.4
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.5.0
Assignee: Giuseppe Scrivano
QA Contact: Sunil Choudhary
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-10-24 18:57 UTC by Weibin Liang
Modified: 2020-07-13 17:12 UTC (History)
5 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-07-13 17:11:31 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift origin pull 25008 0 None closed Bug 1765301: UPSTREAM: 89379: deflake grace period for pod deletion in node e2e 2020-10-07 16:18:53 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:12:02 UTC

Description Weibin Liang 2019-10-24 18:57:50 UTC
Description of problem:
[k8s.io] [sig-node] Pods Extended [k8s.io] Delete Grace Period should be submitted and removed [Conformance] [Suite:openshift/conformance/parallel/minimal] [Suite:k8s] expand_less	48s
fail [k8s.io/kubernetes/test/e2e/node/pods.go:163]: kubelet never observed the termination notice
Unexpected error:
    <*errors.errorString | 0xc000284100>: {
        s: "timed out waiting for the condition",
    }
    timed out waiting for the condition
occurred




Additional info:
https://prow.svc.ci.openshift.org/view/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.3/160

https://testgrid.k8s.io/redhat-openshift-release-4.3-blocking-ocp#release-openshift-ocp-installer-e2e-aws-4.3

Comment 1 Ryan Phillips 2019-11-07 16:20:58 UTC
Discussion about this error here: https://github.com/kubernetes/kubernetes/issues/84610

Comment 2 Peter Hunt 2019-12-12 19:23:09 UTC
Moving this to 4.4 because it is low priority and I haven't reproduced it locally yet.

Comment 4 Peter Hunt 2020-05-20 18:51:36 UTC
Giuseppe, can take this one over?

Comment 5 Giuseppe Scrivano 2020-05-21 10:48:12 UTC
there is already an upstream fix.

Backport here: https://github.com/openshift/origin/pull/25008

Comment 6 Preethi Thomas 2020-05-22 19:00:33 UTC
@Mrunal: What is the next step for this bz?

Comment 9 Sunil Choudhary 2020-06-10 15:38:28 UTC
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.0-0.nightly-2020-06-09-223121   True        False         132m    Cluster version is 4.5.0-0.nightly-2020-06-09-223121

$ ./_output/local/bin/linux/amd64/openshift-tests run-test "[k8s.io] [sig-node] Pods Extended [k8s.io] Delete Grace Period should be submitted and removed [Flaky] [Suite:k8s]"
I0610 19:16:36.811246   29225 test_context.go:423] Tolerating taints "node-role.kubernetes.io/master" when considering if nodes are ready
Jun 10 19:16:36.870: INFO: Waiting up to 30m0s for all (but 100) nodes to be schedulable
Jun 10 19:16:39.173: INFO: Waiting up to 10m0s for all pods (need at least 0) in namespace 'kube-system' to be running and ready
Jun 10 19:16:40.080: INFO: 0 / 0 pods in namespace 'kube-system' are running and ready (0 seconds elapsed)
Jun 10 19:16:40.080: INFO: expected 0 pod replicas in namespace 'kube-system', 0 are Running and Ready.
Jun 10 19:16:40.080: INFO: Waiting up to 5m0s for all daemonsets in namespace 'kube-system' to start
Jun 10 19:16:40.407: INFO: e2e test version: v1.18.0-rc.1
Jun 10 19:16:40.796: INFO: kube-apiserver version: v1.18.3+cc5c3e3
Jun 10 19:16:41.204: INFO: Cluster IP family: ipv4
[BeforeEach] [Top Level]
  /home/sunilc/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/util/framework.go:1374
[BeforeEach] [Top Level]
  /home/sunilc/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/util/framework.go:1374
[BeforeEach] [Top Level]
  /home/sunilc/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/test/extended/util/test.go:58
[BeforeEach] [k8s.io] [sig-node] Pods Extended
  /home/sunilc/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:178
STEP: Creating a kubernetes client
STEP: Building a namespace api object, basename pods
Jun 10 19:16:42.258: INFO: About to run a Kube e2e test, ensuring namespace is privileged
Jun 10 19:16:46.425: INFO: No PodSecurityPolicies found; assuming PodSecurityPolicy is disabled.
STEP: Waiting for a default service account to be provisioned in namespace
[BeforeEach] [k8s.io] Delete Grace Period
  /home/sunilc/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/node/pods.go:50
[It] should be submitted and removed [Flaky] [Suite:k8s]
  /home/sunilc/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/node/pods.go:59
STEP: creating the pod
STEP: setting up selector
STEP: submitting the pod to kubernetes
STEP: verifying the pod is in kubernetes
STEP: deleting the pod gracefully
STEP: verifying the kubelet observed the termination notice
Jun 10 19:17:06.330: INFO: start=2020-06-10 19:16:59.231967197 +0530 IST m=+25.505446802, now=2020-06-10 19:17:06.330768471 +0530 IST m=+32.604248131, kubelet pod: {"metadata":{"name":"pod-submit-remove-e4ee6f98-dc78-4b21-a159-7193ab5552c6","namespace":"e2e-pods-3808","selfLink":"/api/v1/namespaces/e2e-pods-3808/pods/pod-submit-remove-e4ee6f98-dc78-4b21-a159-7193ab5552c6","uid":"b99a01f6-9ca2-484d-a428-d4c74f28b9c6","resourceVersion":"29662","creationTimestamp":"2020-06-10T13:46:47Z","deletionTimestamp":"2020-06-10T13:47:28Z","deletionGracePeriodSeconds":30,"labels":{"name":"foo","time":"743883945"},"annotations":{"k8s.v1.cni.cncf.io/network-status":"[{\n    \"name\": \"openshift-sdn\",\n    \"interface\": \"eth0\",\n    \"ips\": [\n        \"10.128.2.11\"\n    ],\n    \"default\": true,\n    \"dns\": {}\n}]","k8s.v1.cni.cncf.io/networks-status":"[{\n    \"name\": \"openshift-sdn\",\n    \"interface\": \"eth0\",\n    \"ips\": [\n        \"10.128.2.11\"\n    ],\n    \"default\": true,\n    \"dns\": {}\n}]","kubernetes.io/config.seen":"2020-06-10T13:46:47.311024752Z","kubernetes.io/config.source":"api","openshift.io/scc":"anyuid"},"managedFields":[{"manager":"openshift-tests","operation":"Update","apiVersion":"v1","time":"2020-06-10T13:46:47Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:labels":{".":{},"f:name":{},"f:time":{}}},"f:spec":{"f:containers":{"k:{\"name\":\"agnhost\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:resources":{},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{},"f:terminationGracePeriodSeconds":{}}}}]},"spec":{"volumes":[{"name":"default-token-shsq6","secret":{"secretName":"default-token-shsq6","defaultMode":420}}],"containers":[{"name":"agnhost","image":"us.gcr.io/k8s-artifacts-prod/e2e-test-images/agnhost:2.12","args":["pause"],"resources":{},"volumeMounts":[{"name":"default-token-shsq6","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"capabilities":{"drop":["MKNOD"]}}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"ip-10-0-134-0.us-east-2.compute.internal","securityContext":{"seLinuxOptions":{"level":"s0:c24,c9"}},"imagePullSecrets":[{"name":"default-dockercfg-ttn7l"}],"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true},"status":{"phase":"Pending","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-06-10T13:46:47Z"},{"type":"Ready","status":"False","lastProbeTime":null,"lastTransitionTime":"2020-06-10T13:47:00Z","reason":"ContainersNotReady","message":"containers with unready status: [agnhost]"},{"type":"ContainersReady","status":"False","lastProbeTime":null,"lastTransitionTime":"2020-06-10T13:47:00Z","reason":"ContainersNotReady","message":"containers with unready status: [agnhost]"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2020-06-10T13:46:47Z"}],"hostIP":"10.0.134.0","podIP":"10.128.2.11","podIPs":[{"ip":"10.128.2.11"}],"startTime":"2020-06-10T13:46:47Z","containerStatuses":[{"name":"agnhost","state":{"waiting":{"reason":"ContainerCreating"}},"lastState":{},"ready":false,"restartCount":0,"image":"us.gcr.io/k8s-artifacts-prod/e2e-test-images/agnhost:2.12","imageID":"","started":false}],"qosClass":"BestEffort"}}
Jun 10 19:17:10.722: INFO: no pod exists with the name we were looking for, assuming the termination request was observed and completed
[AfterEach] [k8s.io] [sig-node] Pods Extended
  /home/sunilc/go/src/github.com/openshift/origin/_output/local/go/src/github.com/openshift/origin/vendor/k8s.io/kubernetes/test/e2e/framework/framework.go:179
Jun 10 19:17:11.099: INFO: Waiting up to 7m0s for all (but 100) nodes to be ready
STEP: Destroying namespace "e2e-pods-3808" for this suite.
Jun 10 19:17:11.763: INFO: Running AfterSuite actions on all nodes
Jun 10 19:17:11.768: INFO: Running AfterSuite actions on node 1

Comment 11 errata-xmlrpc 2020-07-13 17:11:31 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409


Note You need to log in before you can comment on or make changes to this bug.