This bug was initially created as a copy of Bug #2038019

I am copying this bug because: 



Description of problem:

Static container pod ( create pod yaml file in /etc/kubernetes/manifests/ on worker and let kubelet create the test pod)

met the following error: 

Warning  FailedCreatePodSandBox  <invalid>  kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-212-43.ap-southeast-1.compute.internal_zhouyt_937a3fb4b41f9b5b073f252037c0c642_0(ab9baefc8c6d7bf1664ec01cf81ea37bc76069556d7cedbfaca6150ac1e62541): error adding pod zhouyt_static-web-ip-10-0-212-43.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-212-43.ap-southeast-1.compute.internal/937a3fb4b41f9b5b073f252037c0c642]: expected pod UID "937a3fb4b41f9b5b073f252037c0c642" but got "62ebfd59-5409-4c3e-8c10-069f5b1092e7" from Kube API


Version-Release number of selected component (if applicable):
4.10.0-0.nightly-2022-01-06-183041

How reproducible:
always

Steps to Reproduce:
1. Create pod yaml file in /etc/kubernetes/manifests/ of worker 

$oc debug node/worker1

sh-4.4# cd /etc/kubernetes/manifests/
sh-4.4# cat static-web.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: static-web
  namespace: zhouyt
  labels:
    role: myrole
spec:
  containers:
    - name: web
      image: nginx
      ports:
        - name: web
          containerPort: 80
          protocol: TCP

2. Check the pod cannot be running
3. oc describe pod 

Actual results:

$ oc get pod -n zhouyt -o wide
static-web-ip-10-0-142-146.ap-southeast-1.compute.internal     0/1     ContainerCreating   0          75m   <none>   ip-10-0-142-146.ap-southeast-1.compute.internal   <none>           <none>


$ oc describe pod show the following error: 

  Type     Reason                  Age                   From     Message
  ----     ------                  ----                  ----     -------
  Warning  FailedCreatePodSandBox  77m                   kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(12c7c4a724cea9090dc507c100986355e23eca8c9fdbdc624f9d2f8b7238d751): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API
  Warning  FailedCreatePodSandBox  77m                   kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(d08facb68721d241f66b4b9997fa93be0fb0c76d055085855cb2eb5a749ec1a5): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API
  Warning  FailedCreatePodSandBox  76m                   kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(4831e6b22dbc166257e19725433f4f32f5661ed828e096c45572faf51fb53766): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API
  Warning  FailedCreatePodSandBox  76m                   kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(f194239aefcdcf0aab433a88a8dd3b6a3d455bf505c88fc1efa578a73bc85b83): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API
  Warning  FailedCreatePodSandBox  76m                   kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(c7371edc8aea60be92507837808493eb3403361bbed24d12d726872327e2dd02): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API
  Warning  FailedCreatePodSandBox  76m                   kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(b3e2879ba9967893799448f64f0c7aa5a02f4b4c222a1ba52a2dc9fbe531afd0): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API
  Warning  FailedCreatePodSandBox  76m                   kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(82f37c94e06faf1f1fb862e86e7aed48475f658a533f251a6e1ad6e4a86cf5ab): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API
  Warning  FailedCreatePodSandBox  75m                   kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(7ab12e9f6d606943ee678b0b8400bd4136b07d28acb3a8c65e94772736e0acb4): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API
  Warning  FailedCreatePodSandBox  75m                   kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(6ccb2d7f18c313beb489073bc54a243e67d84403d7888ca942d197f29761f000): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API
  Warning  FailedCreatePodSandBox  2m5s (x336 over 75m)  kubelet  (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(512df204bea415d20618e085ab3dd9cade5816bbf21f49a8b2a7b4e547ccc367): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API

Expected results:


Additional info:

Change the pod to 'hostnetwork' pod. it works well

sh-4.4# cat test-static.yaml
apiVersion: v1
kind: Pod
metadata:
  name: static-hellozy
  namespace: testzy
spec:
  containers:
    - image: "quay.io/openshifttest/hello-pod@sha256:04b6af86b03c1836211be2589db870dba09b7811c197c47c07fbbe33c7f80ef7"
      name: hello-pod
  hostNetwork: true


[root@localhost ~]# oc get po 
NAME                                                             READY   STATUS    RESTARTS   AGE
static-hellozy-ip-10-0-136-177.ap-southeast-1.compute.internal   1/1     Running   0          5s