Bug 1943364 - [sig-api-machinery][Feature:APIServer][Late] kubelet terminates kube-apiserver gracefully [Suite:openshift/conformance/parallel]
Summary: [sig-api-machinery][Feature:APIServer][Late] kubelet terminates kube-apiserve...
Keywords:
Status: CLOSED DUPLICATE of bug 1938353
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 4.7
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Ryan Phillips
QA Contact: Weinan Liu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-03-25 21:49 UTC by Micah Abbott
Modified: 2021-03-31 15:11 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
[sig-api-machinery][Feature:APIServer][Late] kubelet terminates kube-apiserver gracefully [Suite:openshift/conformance/parallel]
Last Closed: 2021-03-31 15:11:44 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Micah Abbott 2021-03-25 21:49:41 UTC
test:
[sig-api-machinery][Feature:APIServer][Late] kubelet terminates kube-apiserver gracefully [Suite:openshift/conformance/parallel] 

is failing frequently in CI, see search results:
https://search.ci.openshift.org/?maxAge=168h&context=1&type=bug%2Bjunit&name=&maxMatches=5&maxBytes=20971520&groupBy=job&search=%5C%5Bsig-api-machinery%5C%5D%5C%5BFeature%3AAPIServer%5C%5D%5C%5BLate%5C%5D+kubelet+terminates+kube-apiserver+gracefully+%5C%5BSuite%3Aopenshift%2Fconformance%2Fparallel%5C%5D

Seen in:

release-openshift-ocp-installer-e2e-aws-serial-4.7
release-openshift-origin-installer-e2e-aws-serial-4.7 
release-openshift-ocp-installer-e2e-metal-serial-4.7

https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.7/1375110641232121856

https://storage.googleapis.com/origin-ci-test/logs/release-openshift-origin-installer-e2e-aws-serial-4.7/1375074844248379392/

https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-metal-serial-4.7/1374948223038263296/

```
I0325 20:48:05.812730   82828 test_context.go:459] Tolerating taints "node-role.kubernetes.io/master" when considering if nodes are ready
[BeforeEach] [Top Level]
  github.com/openshift/origin/test/extended/util/framework.go:1440
[BeforeEach] [Top Level]
  github.com/openshift/origin/test/extended/util/framework.go:1440
[BeforeEach] [Top Level]
  github.com/openshift/origin/test/extended/util/test.go:59
[BeforeEach] [sig-api-machinery][Feature:APIServer][Late]
  github.com/openshift/origin/test/extended/util/client.go:140
STEP: Creating a kubernetes client
[It] kubelet terminates kube-apiserver gracefully [Suite:openshift/conformance/parallel]
  github.com/openshift/origin/test/extended/apiserver/graceful_termination.go:20
[AfterEach] [sig-api-machinery][Feature:APIServer][Late]
  github.com/openshift/origin/test/extended/util/client.go:138
[AfterEach] [sig-api-machinery][Feature:APIServer][Late]
  github.com/openshift/origin/test/extended/util/client.go:139
fail [github.com/onsi/ginkgo.0-origin.0+incompatible/internal/leafnodes/runner.go:64]: kube-apiserver reports a non-graceful termination: v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kube-apiserver-ip-10-0-199-34.us-west-1.compute.internal.166faee8cf67a34d", GenerateName:"", Namespace:"openshift-kube-apiserver", SelfLink:"/api/v1/namespaces/openshift-kube-apiserver/events/kube-apiserver-ip-10-0-199-34.us-west-1.compute.internal.166faee8cf67a34d", UID:"7566e415-df51-4065-9ed7-41855baae3b4", ResourceVersion:"26259", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63752299906, loc:(*time.Location)(0x94eeb80)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"watch-termination", Operation:"Update", APIVersion:"v1", Time:(*v1.Time)(0xc001bf9bc0), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc001bf9be0)}}}, InvolvedObject:v1.ObjectReference{Kind:"Pod", Namespace:"openshift-kube-apiserver", Name:"kube-apiserver-ip-10-0-199-34.us-west-1.compute.internal", UID:"", APIVersion:"v1", ResourceVersion:"", FieldPath:""}, Reason:"NonGracefulTermination", Message:"Previous pod kube-apiserver-ip-10-0-199-34.us-west-1.compute.internal started at 2021-03-25 20:08:56.834108134 +0000 UTC did not terminate gracefully", Source:v1.EventSource{Component:"apiserver", Host:"ip-10-0-199-34"}, FirstTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63752299906, loc:(*time.Location)(0x94eeb80)}}, LastTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63752299906, loc:(*time.Location)(0x94eeb80)}}, Count:1, Type:"Warning", EventTime:v1.MicroTime{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}. Probably kubelet or CRI-O is not giving the time to cleanly shut down. This can lead to connection refused and network I/O timeout errors in other components.

failed: (700ms) 2021-03-25T20:48:06 "[sig-api-machinery][Feature:APIServer][Late] kubelet terminates kube-apiserver gracefully [Suite:openshift/conformance/parallel]"
```

Comment 1 Ryan Phillips 2021-03-31 15:11:44 UTC
4.7 has not been getting all the backports to fix this. We are working in 4.8 to get this more stabilized.

*** This bug has been marked as a duplicate of bug 1938353 ***


Note You need to log in before you can comment on or make changes to this bug.