Bug 2038019

Summary:	static container pod cannot be running due to get different pod UID
Product:	OpenShift Container Platform	Reporter:	zhaozhanqi <zzhao>
Component:	Networking	Assignee:	Tomofumi Hayashi <tohayash>
Networking sub component:	multus	QA Contact:	Weibin Liang <weliang>
Status:	CLOSED WONTFIX	Docs Contact:
Severity:	high
Priority:	high	CC:	sninganu, tohayash
Version:	4.10
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:
Clones:	2066838 (view as bug list)		Environment:
Last Closed:	2024-04-30 18:04:53 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:
Bug Blocks:	2038106, 2066838

Description zhaozhanqi 2022-01-07 04:36:32 UTC

Description of problem:

Static container pod ( create pod yaml file in /etc/kubernetes/manifests/ on worker and let kubelet create the test pod)

met the following error: 

Warning  FailedCreatePodSandBox  <invalid>  kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-212-43.ap-southeast-1.compute.internal_zhouyt_937a3fb4b41f9b5b073f252037c0c642_0(ab9baefc8c6d7bf1664ec01cf81ea37bc76069556d7cedbfaca6150ac1e62541): error adding pod zhouyt_static-web-ip-10-0-212-43.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-212-43.ap-southeast-1.compute.internal/937a3fb4b41f9b5b073f252037c0c642]: expected pod UID "937a3fb4b41f9b5b073f252037c0c642" but got "62ebfd59-5409-4c3e-8c10-069f5b1092e7" from Kube API


Version-Release number of selected component (if applicable):
4.10.0-0.nightly-2022-01-06-183041

How reproducible:
always

Steps to Reproduce:
1. Create pod yaml file in /etc/kubernetes/manifests/ of worker 

$oc debug node/worker1

sh-4.4# cd /etc/kubernetes/manifests/
sh-4.4# cat static-web.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: static-web
  namespace: zhouyt
  labels:
    role: myrole
spec:
  containers:
    - name: web
      image: nginx
      ports:
        - name: web
          containerPort: 80
          protocol: TCP

2. Check the pod cannot be running
3. oc describe pod 

Actual results:

$ oc get pod -n zhouyt -o wide
static-web-ip-10-0-142-146.ap-southeast-1.compute.internal     0/1     ContainerCreating   0          75m   <none>   ip-10-0-142-146.ap-southeast-1.compute.internal   <none>           <none>


$ oc describe pod show the following error: 

  Type     Reason                  Age                   From     Message
  ----     ------                  ----                  ----     -------
  Warning  FailedCreatePodSandBox  77m                   kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(12c7c4a724cea9090dc507c100986355e23eca8c9fdbdc624f9d2f8b7238d751): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API
  Warning  FailedCreatePodSandBox  77m                   kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(d08facb68721d241f66b4b9997fa93be0fb0c76d055085855cb2eb5a749ec1a5): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API
  Warning  FailedCreatePodSandBox  76m                   kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(4831e6b22dbc166257e19725433f4f32f5661ed828e096c45572faf51fb53766): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API
  Warning  FailedCreatePodSandBox  76m                   kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(f194239aefcdcf0aab433a88a8dd3b6a3d455bf505c88fc1efa578a73bc85b83): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API
  Warning  FailedCreatePodSandBox  76m                   kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(c7371edc8aea60be92507837808493eb3403361bbed24d12d726872327e2dd02): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API
  Warning  FailedCreatePodSandBox  76m                   kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(b3e2879ba9967893799448f64f0c7aa5a02f4b4c222a1ba52a2dc9fbe531afd0): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API
  Warning  FailedCreatePodSandBox  76m                   kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(82f37c94e06faf1f1fb862e86e7aed48475f658a533f251a6e1ad6e4a86cf5ab): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API
  Warning  FailedCreatePodSandBox  75m                   kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(7ab12e9f6d606943ee678b0b8400bd4136b07d28acb3a8c65e94772736e0acb4): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API
  Warning  FailedCreatePodSandBox  75m                   kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(6ccb2d7f18c313beb489073bc54a243e67d84403d7888ca942d197f29761f000): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API
  Warning  FailedCreatePodSandBox  2m5s (x336 over 75m)  kubelet  (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal_zhouyt_70a2d08f8ad8cd5dfc51ad71fefd1e78_0(512df204bea415d20618e085ab3dd9cade5816bbf21f49a8b2a7b4e547ccc367): error adding pod zhouyt_static-web-ip-10-0-142-146.ap-southeast-1.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [zhouyt/static-web-ip-10-0-142-146.ap-southeast-1.compute.internal/70a2d08f8ad8cd5dfc51ad71fefd1e78]: expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API

Expected results:


Additional info:

Change the pod to 'hostnetwork' pod. it works well

sh-4.4# cat test-static.yaml
apiVersion: v1
kind: Pod
metadata:
  name: static-hellozy
  namespace: testzy
spec:
  containers:
    - image: "quay.io/openshifttest/hello-pod@sha256:04b6af86b03c1836211be2589db870dba09b7811c197c47c07fbbe33c7f80ef7"
      name: hello-pod
  hostNetwork: true


[root@localhost ~]# oc get po 
NAME                                                             READY   STATUS    RESTARTS   AGE
static-hellozy-ip-10-0-136-177.ap-southeast-1.compute.internal   1/1     Running   0          5s

Comment 2 zhaozhanqi 2022-01-07 04:39:58 UTC

source like related this PR https://github.com/k8snetworkplumbingwg/multus-cni/pull/742/files

Comment 3 Douglas Smith 2022-01-07 20:52:18 UTC

Thanks for the report. I've been able to successfully reproduce, and I'm pending a conversation with my team about how to handle this scenario. Thanks.

Upstream issue also filed @ https://github.com/k8snetworkplumbingwg/multus-cni/issues/773

Changing to high priority as this prevents most typical static pods from being created.

Comment 4 Tomofumi Hayashi 2022-01-18 04:41:00 UTC

Upstream PR: https://github.com/k8snetworkplumbingwg/multus-cni/pull/775

Comment 7 Weibin Liang 2022-03-30 19:58:28 UTC

https://github.com/openshift/multus-cni/pull/121 merged in https://amd64.ocp.releases.ci.openshift.org/releasestream/4.11.0-0.nightly/release/4.11.0-0.nightly-2022-03-29-152521, but verification failed in this build.

[weliang@weliang verification-tests]$ oc get pod
NAME                                                   READY   STATUS              RESTARTS   AGE
static-web-ip-10-0-217-25.us-east-2.compute.internal   0/1     ContainerCreating   0          2m3s
[weliang@weliang verification-tests]$ 
[weliang@weliang verification-tests]$ 
[weliang@weliang verification-tests]$ oc describe pod static-web-ip-10-0-217-25.us-east-2.compute.internal
Name:           static-web-ip-10-0-217-25.us-east-2.compute.internal
Namespace:      test
Priority:       0
Node:           ip-10-0-217-25.us-east-2.compute.internal/10.0.217.25
Start Time:     Wed, 30 Mar 2022 15:38:47 -0400
Labels:         role=myrole
Annotations:    k8s.ovn.org/pod-networks:
                  {"default":{"ip_addresses":["10.129.2.10/23"],"mac_address":"0a:58:0a:81:02:0a","gateway_ips":["10.129.2.1"],"ip_address":"10.129.2.10/23"...
                kubernetes.io/config.hash: ab116838dc61a9295b25c04ab60fe03c
                kubernetes.io/config.mirror: ab116838dc61a9295b25c04ab60fe03c
                kubernetes.io/config.seen: 2022-03-30T19:38:47.329193012Z
                kubernetes.io/config.source: file
                openshift.io/scc: restricted
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  Node/ip-10-0-217-25.us-east-2.compute.internal
Containers:
  web:
    Container ID:   
    Image:          nginx
    Image ID:       
    Port:           80/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:         <none>
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:            <none>
QoS Class:          BestEffort
Node-Selectors:     <none>
Tolerations:        :NoExecute op=Exists
Events:
  Type     Reason                  Age   From     Message
  ----     ------                  ----  ----     -------
  Warning  FailedCreatePodSandBox  2m7s  kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-217-25.us-east-2.compute.internal_test_ab116838dc61a9295b25c04ab60fe03c_0(65c1d4dced2305c592eb465b65228435a98d1d5c39600deb97b9f2a2f0f4e2b1): error adding pod test_static-web-ip-10-0-217-25.us-east-2.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [test/static-web-ip-10-0-217-25.us-east-2.compute.internal/ab116838dc61a9295b25c04ab60fe03c:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[test/static-web-ip-10-0-217-25.us-east-2.compute.internal 65c1d4dced2305c592eb465b65228435a98d1d5c39600deb97b9f2a2f0f4e2b1] [test/static-web-ip-10-0-217-25.us-east-2.compute.internal 65c1d4dced2305c592eb465b65228435a98d1d5c39600deb97b9f2a2f0f4e2b1] pod deleted before sandbox ADD operation began
'
  Warning  FailedCreatePodSandBox  2m5s  kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-217-25.us-east-2.compute.internal_test_ab116838dc61a9295b25c04ab60fe03c_0(8ed13112f26159f599f7de0a5ef39e347c64a17e1af181171e4ba33caf3ef62c): error adding pod test_static-web-ip-10-0-217-25.us-east-2.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [test/static-web-ip-10-0-217-25.us-east-2.compute.internal/ab116838dc61a9295b25c04ab60fe03c:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[test/static-web-ip-10-0-217-25.us-east-2.compute.internal 8ed13112f26159f599f7de0a5ef39e347c64a17e1af181171e4ba33caf3ef62c] [test/static-web-ip-10-0-217-25.us-east-2.compute.internal 8ed13112f26159f599f7de0a5ef39e347c64a17e1af181171e4ba33caf3ef62c] pod deleted before sandbox ADD operation began
'
  Warning  FailedCreatePodSandBox  111s  kubelet  Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_static-web-ip-10-0-217-25.us-east-2.compute.internal_test_ab116838dc61a9295b25c04ab60fe03c_0(a6bb9badcb97fdc1628d3ee2e1f098084dbf028a7dbfa3f73ac2d6514720ec6d): error adding pod test_static-web-ip-10-0-217-25.us-east-2.compute.internal to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [test/static-web-ip-10-0-217-25.us-east-2.compute.internal/ab116838dc61a9295b25c04ab60fe03c:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[test/static-web-ip-10-0-217-25.us-east-2.compute.internal a6bb9badcb97fdc1628d3ee2e1f098084dbf028a7dbfa3f73ac2d6514720ec6d] [test/static-web-ip-10-0-217-25.us-east-2.compute.internal a6bb9badcb97fdc1628d3ee2e1f098084dbf028a7dbfa3f73ac2d6514720ec6d] pod deleted before sandbox ADD operation began
'

Comment 10 Weibin Liang 2022-12-19 15:56:42 UTC

The above verification did not see the error message like - "expected pod UID "70a2d08f8ad8cd5dfc51ad71fefd1e78" but got "d59103af-b6f6-4e90-84ba-d91112074703" from Kube API".

Will open a new bug for the new error message of "CNI request failed with status 400".

Comment 11 Rory Thrasher 2024-04-30 18:04:53 UTC

OCP is no longer using Bugzilla and this bug appears to have been left in an orphaned state. If the bug is still relevant, please open a new issue in the OCPBUGS Jira project: https://issues.redhat.com/projects/OCPBUGS/summary