Description of problem: Static container pod ( create pod yaml file in /etc/kubernetes/manifests/ on worker and let kubelet create the test pod) met the following error: Warning FailedCreatePodSandBox <invalid> kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-212-43.ap-southeast-1.compute.internal_zhouyt_937a3fb4b41f9b5b073f252037c0c642_0(ab9baefc8c6d7bf1664ec01cf81ea37bc76069556d7cedbfaca6150ac1e62541): error adding pod zhouyt_static-web-ip-10-0-212-43.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-212-43.ap-southeast-1.compute.internal/937a3fb4b41f9b5b073f252037c0c642]: expected pod UID "937a3fb4b41f9b5b073f252037c0c642" but got "62ebfd59-5409-4c3e-8c10-069f5b1092e7" from Kube API Version-Release number of selected component (if applicable): 4.10.0-0.nightly-2022-01-06-183041 How reproducible: always Steps to Reproduce: 1. Create pod yaml file in /etc/kubernetes/manifests/ of worker $oc debug node/worker1 sh-4.4# cd /etc/kubernetes/manifests/ sh-4.4# cat static-web.yaml apiVersion: v1 kind: Pod metadata: name: static-web namespace: zhouyt labels: role: myrole spec: containers: - name: web image: nginx ports: - name: web containerPort: 80 protocol: TCP 2. Check the pod cannot be running 3. oc describe pod Actual results: $ oc get pod -n zhouyt -o wide static-web-ip-10-0-142-146.ap-southeast-1.compute.internal 0/1 ContainerCreating 0 75m <none> ip-10-0-142-146.ap-southeast-1.compute.internal <none> <none> $ oc describe pod show the following error: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreatePodSandBox 77m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(12c7c4a724cea9090dc507c100986355e23eca8c9fdbdc624f9d2f8b7238d751): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API Warning FailedCreatePodSandBox 77m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(d08facb68721d241f66b4b9997fa93be0fb0c76d055085855cb2eb5a749ec1a5): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API Warning FailedCreatePodSandBox 76m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(4831e6b22dbc166257e19725433f4f32f5661ed828e096c45572faf51fb53766): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API Warning FailedCreatePodSandBox 76m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(f194239aefcdcf0aab433a88a8dd3b6a3d455bf505c88fc1efa578a73bc85b83): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API Warning FailedCreatePodSandBox 76m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(c7371edc8aea60be92507837808493eb3403361bbed24d12d726872327e2dd02): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API Warning FailedCreatePodSandBox 76m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(b3e2879ba9967893799448f64f0c7aa5a02f4b4c222a1ba52a2dc9fbe531afd0): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API Warning FailedCreatePodSandBox 76m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(82f37c94e06faf1f1fb862e86e7aed48475f658a533f251a6e1ad6e4a86cf5ab): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API Warning FailedCreatePodSandBox 75m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(7ab12e9f6d606943ee678b0b8400bd4136b07d28acb3a8c65e94772736e0acb4): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API Warning FailedCreatePodSandBox 75m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(6ccb2d7f18c313beb489073bc54a243e67d84403d7888ca942d197f29761f000): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API Warning FailedCreatePodSandBox 2m5s (x336 over 75m) kubelet (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(512df204bea415d20618e085ab3dd9cade5816bbf21f49a8b2a7b4e547ccc367): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API Expected results: Additional info: Change the pod to 'hostnetwork' pod. it works well sh-4.4# cat test-static.yaml apiVersion: v1 kind: Pod metadata: name: static-hellozy namespace: testzy spec: containers: - image: "quay.io/openshifttest/hello-pod@sha256:04b6af86b03c1836211be2589db870dba09b7811c197c47c07fbbe33c7f80ef7" name: hello-pod hostNetwork: true [root@localhost ~]# oc get po NAME READY STATUS RESTARTS AGE static-hellozy-ip-10-0-136-177.ap-southeast-1.compute.internal 1/1 Running 0 5s
source like related this PR https://github.com/k8snetworkplumbingwg/multus-cni/pull/742/files
Thanks for the report. I've been able to successfully reproduce, and I'm pending a conversation with my team about how to handle this scenario. Thanks. Upstream issue also filed @ https://github.com/k8snetworkplumbingwg/multus-cni/issues/773 Changing to high priority as this prevents most typical static pods from being created.
Upstream PR: https://github.com/k8snetworkplumbingwg/multus-cni/pull/775
https://github.com/openshift/multus-cni/pull/121 merged in https://amd64.ocp.releases.ci.openshift.org/releasestream/4.11.0-0.nightly/release/4.11.0-0.nightly-2022-03-29-152521, but verification failed in this build. [weliang@weliang verification-tests]$ oc get pod NAME READY STATUS RESTARTS AGE static-web-ip-10-0-217-25.us-east-2.compute.internal 0/1 ContainerCreating 0 2m3s [weliang@weliang verification-tests]$ [weliang@weliang verification-tests]$ [weliang@weliang verification-tests]$ oc describe pod static-web-ip-10-0-217-25.us-east-2.compute.internal Name: static-web-ip-10-0-217-25.us-east-2.compute.internal Namespace: test Priority: 0 Node: ip-10-0-217-25.us-east-2.compute.internal/10.0.217.25 Start Time: Wed, 30 Mar 2022 15:38:47 -0400 Labels: role=myrole Annotations: k8s.ovn.org/pod-networks: {"default":{"ip_addresses":["10.129.2.10/23"],"mac_address":"0a:58:0a:81:02:0a","gateway_ips":["10.129.2.1"],"ip_address":"10.129.2.10/23"... kubernetes.io/config.hash: ab116838dc61a9295b25c04ab60fe03c kubernetes.io/config.mirror: ab116838dc61a9295b25c04ab60fe03c kubernetes.io/config.seen: 2022-03-30T19:38:47.329193012Z kubernetes.io/config.source: file openshift.io/scc: restricted Status: Pending IP: IPs: <none> Controlled By: Node/ip-10-0-217-25.us-east-2.compute.internal Containers: web: Container ID: Image: nginx Image ID: Port: 80/TCP Host Port: 0/TCP State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment: <none> Mounts: <none> Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: <none> QoS Class: BestEffort Node-Selectors: <none> Tolerations: :NoExecute op=Exists Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreatePodSandBox 2m7s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-217-25.us-east-2.compute.internal_test_ab116838dc61a9295b25c04ab60fe03c_0(65c1d4dced2305c592eb465b65228435a98d1d5c39600deb97b9f2a2f0f4e2b1): error adding pod test_static-web-ip-10-0-217-25.us-east-2.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [test/static-web-ip-10-0-217-25.us-east-2.compute.internal/ab116838dc61a9295b25c04ab60fe03c:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[test/static-web-ip-10-0-217-25.us-east-2.compute.internal 65c1d4dced2305c592eb465b65228435a98d1d5c39600deb97b9f2a2f0f4e2b1] [test/static-web-ip-10-0-217-25.us-east-2.compute.internal 65c1d4dced2305c592eb465b65228435a98d1d5c39600deb97b9f2a2f0f4e2b1] pod deleted before sandbox ADD operation began ' Warning FailedCreatePodSandBox 2m5s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-217-25.us-east-2.compute.internal_test_ab116838dc61a9295b25c04ab60fe03c_0(8ed13112f26159f599f7de0a5ef39e347c64a17e1af181171e4ba33caf3ef62c): error adding pod test_static-web-ip-10-0-217-25.us-east-2.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [test/static-web-ip-10-0-217-25.us-east-2.compute.internal/ab116838dc61a9295b25c04ab60fe03c:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[test/static-web-ip-10-0-217-25.us-east-2.compute.internal 8ed13112f26159f599f7de0a5ef39e347c64a17e1af181171e4ba33caf3ef62c] [test/static-web-ip-10-0-217-25.us-east-2.compute.internal 8ed13112f26159f599f7de0a5ef39e347c64a17e1af181171e4ba33caf3ef62c] pod deleted before sandbox ADD operation began ' Warning FailedCreatePodSandBox 111s kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-217-25.us-east-2.compute.internal_test_ab116838dc61a9295b25c04ab60fe03c_0(a6bb9badcb97fdc1628d3ee2e1f098084dbf028a7dbfa3f73ac2d6514720ec6d): error adding pod test_static-web-ip-10-0-217-25.us-east-2.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [test/static-web-ip-10-0-217-25.us-east-2.compute.internal/ab116838dc61a9295b25c04ab60fe03c:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[test/static-web-ip-10-0-217-25.us-east-2.compute.internal a6bb9badcb97fdc1628d3ee2e1f098084dbf028a7dbfa3f73ac2d6514720ec6d] [test/static-web-ip-10-0-217-25.us-east-2.compute.internal a6bb9badcb97fdc1628d3ee2e1f098084dbf028a7dbfa3f73ac2d6514720ec6d] pod deleted before sandbox ADD operation began '
The above verification did not see the error message like - "expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API". Will open a new bug for the new error message of "CNI request failed with status 400".
OCP is no longer using Bugzilla and this bug appears to have been left in an orphaned state. If the bug is still relevant, please open a new issue in the OCPBUGS Jira project: https://issues.redhat.com/projects/OCPBUGS/summary