Bug 2098587
| Summary: | CNI request failed with status 400 - failed to get pod annotation: timed out waiting for annotations | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | sunzhaohua <zhsun> |
| Component: | Networking | Assignee: | Miguel Duarte Barroso <mduarted> |
| Networking sub component: | ovn-kubernetes | QA Contact: | Anurag saxena <anusaxen> |
| Status: | CLOSED DUPLICATE | Docs Contact: | |
| Severity: | medium | ||
| Priority: | medium | CC: | akaris, ffernand |
| Version: | 4.11 | ||
| Target Milestone: | --- | ||
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2022-06-22 08:18:10 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I think we're stuck in a loop, but unfortunately the provided must gather does not feature the ovn-kube-node CNI logs - meaning we do not have the full picture. Never the less, there's enough information to hint with a high degree of confidence at what it happening: - the kubelet-killer pod is scheduled to the zhsunali1-tmw46-worker-us-east-1b-4tpq9 node (in its spec) - pod kills kubelet - the node is torn down by the machine-api and reprovisioned - node comes back online, kubelet-killer pod is scheduled to it and immediately kills kubelet The above happens in a loop. Furthermore, the kubelet-killer pod is never reported as Successfully bound pod to node by the kubescheduler, because it kills kubelet when it starts, so kubelet can't report its status. About the annotations (reason why the visible events are being printed): - when a new pod is scheduled, it will always feature the same name - `openshift-machine-api/kubelet-killer` - meaning while they are different entities, to ovnk, they are the same. - then the new pod is instantiated, it will find this stale key `openshift-machine-api/kubelet-killer`, and will attempt to clean it up. - when ovnk-master tries to delete the stale logical port, it fails to do so because it cannot find the OVN annotations on the pod. - this operation is retried without limit, thus causing the ovn-kube-master to timeout attempting to set the annotations. Two different PRs were recently merged that improve this behavior: - https://github.com/openshift/ovn-kubernetes/pull/1123 - https://github.com/openshift/ovn-kubernetes/pull/1147 These 2 PRs are available on the latest nightly - 4.11.0-0.nightly-2022-06-21-040754 . Could you re-try your test with that nightly build ? Thanks mduarted, I re-tried, this works well on 4.11.0-0.nightly-2022-06-21-151125. $ oc get node NAME STATUS ROLES AGE VERSION zhsunali2-fz5ld-master-0 Ready master 3h48m v1.24.0+284d62a zhsunali2-fz5ld-master-1 Ready master 3h48m v1.24.0+284d62a zhsunali2-fz5ld-master-2 Ready master 3h50m v1.24.0+284d62a zhsunali2-fz5ld-worker-us-east-1a-84njt Ready worker 3h30m v1.24.0+284d62a zhsunali2-fz5ld-worker-us-east-1b-kn6vl NotReady worker 35m v1.24.0+284d62a Thank you for re-trying, and posting back the results. Closing as a duplicate; feel free to re-open the bug should it occur again. *** This bug has been marked as a duplicate of bug 2092579 *** |
Description of problem: When create a pod on Alicloud ovn cluster to kill kubelet on the node, node can’t become NotReady Version-Release number of selected component (if applicable): 4.11.0-0.nightly-2022-06-15-222801 How reproducible: Always Steps to Reproduce: 1. Create a pod to kill kubelet apiVersion: v1 kind: Pod metadata: labels: kubelet-killer: "" name: kubelet-killer namespace: openshift-machine-api spec: containers: - command: - pkill - -STOP - kubelet image: quay.io/openshifttest/base-alpine@sha256:0b379877aba876774e0043ea5ba41b0c574825ab910d32b43c05926fab4eea22 imagePullPolicy: Always name: kubelet-killer securityContext: privileged: true enableServiceLinks: true hostPID: true nodeName: zhsunali1-tmw46-worker-us-east-1b-4tpq9 restartPolicy: Never 2. 3. Actual results: $ oc get po NAME READY STATUS RESTARTS AGE cluster-autoscaler-operator-665454c877-5b82q 2/2 Running 0 4h53m cluster-baremetal-operator-79d6c7b5c9-bltk9 2/2 Running 0 4h53m kubelet-killer 0/1 ContainerCreating 0 50m $ oc get event 61s Warning FailedCreatePodSandBox pod/kubelet-killer (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kubelet-killer_openshift-machine-api_3d005c58-145b-4f97-9386-dee9745aae60_0(d2e4580fa97f78d64e9e3cf46d63fa369fe2188b63f2d0d0d48f80976f43c26f): error adding pod openshift-machine-api_kubelet-killer to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [openshift-machine-api/kubelet-killer/3d005c58-145b-4f97-9386-dee9745aae60:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[openshift-machine-api/kubelet-killer d2e4580fa97f78d64e9e3cf46d63fa369fe2188b63f2d0d0d48f80976f43c26f] [openshift-machine-api/kubelet-killer d2e4580fa97f78d64e9e3cf46d63fa369fe2188b63f2d0d0d48f80976f43c26f] failed to get pod annotation: timed out waiting for annotations: context deadline exceeded... $ oc get node NAME STATUS ROLES AGE VERSION zhsunali1-tmw46-master-0 Ready master 5h19m v1.24.0+cb71478 zhsunali1-tmw46-master-1 Ready master 5h19m v1.24.0+cb71478 zhsunali1-tmw46-master-2 Ready master 5h17m v1.24.0+cb71478 zhsunali1-tmw46-worker-us-east-1a-td5wz Ready worker 5h5m v1.24.0+cb71478 zhsunali1-tmw46-worker-us-east-1b-4tpq9 Ready worker 4h32m v1.24.0+cb71478 $ oc debug node/zhsunali1-tmw46-worker-us-east-1b-4tpq9 [15:20:56] Starting pod/zhsunali1-tmw46-worker-us-east-1b-4tpq9-debug ... To use host binaries, run `chroot /host` chroot /host Pod IP: 10.0.107.42 If you don't see a command prompt, try pressing enter. sh-4.4# chroot /host sh-4.4# systemctl status kubelet ● kubelet.service - Kubernetes Kubelet Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/kubelet.service.d └─10-mco-default-env.conf, 10-mco-default-madv.conf, 20-alibaba-node-name.conf, 20-logging.conf Active: active (running) since Mon 2022-06-20 02:32:46 UTC; 4h 48min ago Expected results: Node can become NotReady Additional info: must-gather: https://file.rdu.redhat.com/~zhsun/must-gather.local.653451416107662964.zip