Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2100642

Summary: [SDN-OVN-Migration] Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox
Product: OpenShift Container Platform Reporter: Weibin Liang <weliang>
Component: NetworkingAssignee: Nikhil Simha <nsimha>
Networking sub component: multus QA Contact: Weibin Liang <weliang>
Status: CLOSED DUPLICATE Docs Contact:
Severity: high    
Priority: high CC: dosmith
Version: 4.11   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-07-07 17:55:11 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Weibin Liang 2022-06-23 20:52:19 UTC
Description of problem:
Multus pod got stuck at ContainerCreating state after SDN to OVN migration.

Version-Release number of selected component (if applicable):
4.11.0-0.nightly-2022-06-23-092832

How reproducible:
Always

Steps to Reproduce:
1. Create a multus pod
[weliang@weliang SDN-OVN-Migration]$ oc project multus-upgrade
Now using project "multus-upgrade" on server "https://api.weliang-6237.qe.gcp.devcluster.openshift.com:6443".
[weliang@weliang SDN-OVN-Migration]$ oc get all
NAME                                        READY   STATUS    RESTARTS   AGE
pod/macvlan-bridge-whereabouts-pod1-5gjxr   1/1     Running   0          5m40s

NAME                                                    DESIRED   CURRENT   READY   AGE
replicationcontroller/macvlan-bridge-whereabouts-pod1   1         1         1       5m40s
[weliang@weliang SDN-OVN-Migration]$ oc describe pod/macvlan-bridge-whereabouts-pod1-5gjxr
Name:         macvlan-bridge-whereabouts-pod1-5gjxr
Namespace:    multus-upgrade
Priority:     0
Node:         weliang-6237-llpdj-worker-c-s6blp.c.openshift-qe.internal/10.0.128.4
Start Time:   Thu, 23 Jun 2022 15:57:24 -0400
Labels:       name=test-pod1
Annotations:  k8s.v1.cni.cncf.io/network-status:
                [{
                    "name": "openshift-sdn",
                    "interface": "eth0",
                    "ips": [
                        "10.131.0.16"
                    ],
                    "default": true,
                    "dns": {}
                },{
                    "name": "multus-upgrade/macvlan-bridge-whereabouts",
                    "interface": "net1",
                    "ips": [
                        "192.168.22.101"
                    ],
                    "mac": "0e:a0:da:df:13:1a",
                    "dns": {}
                }]
              k8s.v1.cni.cncf.io/networks: macvlan-bridge-whereabouts
              k8s.v1.cni.cncf.io/networks-status:
                [{
                    "name": "openshift-sdn",
                    "interface": "eth0",
                    "ips": [
                        "10.131.0.16"
                    ],
                    "default": true,
                    "dns": {}
                },{
                    "name": "multus-upgrade/macvlan-bridge-whereabouts",
                    "interface": "net1",
                    "ips": [
                        "192.168.22.101"
                    ],
                    "mac": "0e:a0:da:df:13:1a",
                    "dns": {}
                }]
              openshift.io/scc: restricted-v2
              seccomp.security.alpha.kubernetes.io/pod: runtime/default
Status:       Running
IP:           10.131.0.16
IPs:
  IP:           10.131.0.16
Controlled By:  ReplicationController/macvlan-bridge-whereabouts-pod1
Containers:
  macvlan-bridge-whereabouts:
    Container ID:   cri-o://8cb5d3f2081fb94a11d8024b32ca7966e231a5b52c37a677c4af7a6444aeac40
    Image:          quay.io/openshifttest/hello-sdn@sha256:2af5b5ec480f05fda7e9b278023ba04724a3dd53a296afcd8c13f220dec52197
    Image ID:       quay.io/openshifttest/hello-sdn@sha256:2af5b5ec480f05fda7e9b278023ba04724a3dd53a296afcd8c13f220dec52197
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Thu, 23 Jun 2022 15:57:31 -0400
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-7mj8z (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  kube-api-access-7mj8z:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    ConfigMapName:           openshift-service-ca.crt
    ConfigMapOptional:       <nil>
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason          Age    From               Message
  ----    ------          ----   ----               -------
  Normal  Scheduled       5m49s  default-scheduler  Successfully assigned multus-upgrade/macvlan-bridge-whereabouts-pod1-5gjxr to weliang-6237-llpdj-worker-c-s6blp.c.openshift-qe.internal by weliang-6237-llpdj-master-1.c.openshift-qe.internal
  Normal  AddedInterface  5m48s  multus             Add eth0 [10.131.0.16/23] from openshift-sdn
  Normal  AddedInterface  5m47s  multus             Add net1 [192.168.22.101/30] from multus-upgrade/macvlan-bridge-whereabouts
  Normal  Pulling         5m47s  kubelet            Pulling image "quay.io/openshifttest/hello-sdn@sha256:2af5b5ec480f05fda7e9b278023ba04724a3dd53a296afcd8c13f220dec52197"
  Normal  Pulled          5m43s  kubelet            Successfully pulled image "quay.io/openshifttest/hello-sdn@sha256:2af5b5ec480f05fda7e9b278023ba04724a3dd53a296afcd8c13f220dec52197" in 4.099360351s
  Normal  Created         5m43s  kubelet            Created container macvlan-bridge-whereabouts
  Normal  Started         5m43s  kubelet            Started container macvlan-bridge-whereabouts
[weliang@weliang SDN-OVN-Migration]$ oc get replicationcontroller/macvlan-bridge-whereabouts-pod1 -o yaml
apiVersion: v1
kind: ReplicationController
metadata:
  creationTimestamp: "2022-06-23T19:57:24Z"
  generation: 1
  labels:
    name: test-pod
  name: macvlan-bridge-whereabouts-pod1
  namespace: multus-upgrade
  resourceVersion: "37632"
  uid: c26d315e-2380-434c-8e55-67281bcee598
spec:
  replicas: 1
  selector:
    name: test-pod1
  template:
    metadata:
      annotations:
        k8s.v1.cni.cncf.io/networks: macvlan-bridge-whereabouts
      creationTimestamp: null
      labels:
        name: test-pod1
    spec:
      containers:
      - image: quay.io/openshifttest/hello-sdn@sha256:2af5b5ec480f05fda7e9b278023ba04724a3dd53a296afcd8c13f220dec52197
        imagePullPolicy: IfNotPresent
        name: macvlan-bridge-whereabouts
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
status:
  availableReplicas: 1
  fullyLabeledReplicas: 1
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
[weliang@weliang SDN-OVN-Migration]$ 

2. Run migration by using https://github.com/openshift/release/blob/master/ci-operator/step-registry/ovn/sdn-migration/ovn-sdn-migration-commands.sh

3. Migration succeed, check multus pod again after migration
[weliang@weliang SDN-OVN-Migration]$ oc get all
NAME                                        READY   STATUS              RESTARTS   AGE
pod/macvlan-bridge-whereabouts-pod1-5gdkz   0/1     ContainerCreating   0          35m

NAME                                                    DESIRED   CURRENT   READY   AGE
replicationcontroller/macvlan-bridge-whereabouts-pod1   1         1         0       52m
[weliang@weliang SDN-OVN-Migration]$ 
[weliang@weliang SDN-OVN-Migration]$ oc get net-attach-def
NAME                         AGE
macvlan-bridge-whereabouts   52m
[weliang@weliang SDN-OVN-Migration]$ oc describe pod/macvlan-bridge-whereabouts-pod1-5gdkz
Name:           macvlan-bridge-whereabouts-pod1-5gdkz
Namespace:      multus-upgrade
Priority:       0
Node:           weliang-6237-llpdj-worker-b-2kg9p.c.openshift-qe.internal/10.0.128.3
Start Time:     Thu, 23 Jun 2022 16:14:30 -0400
Labels:         name=test-pod1
Annotations:    k8s.ovn.org/pod-networks:
                  {"default":{"ip_addresses":["10.130.0.16/23"],"mac_address":"0a:58:0a:82:00:10","gateway_ips":["10.130.0.1"],"ip_address":"10.130.0.16/23"...
                k8s.v1.cni.cncf.io/networks: macvlan-bridge-whereabouts
                openshift.io/scc: restricted-v2
                seccomp.security.alpha.kubernetes.io/pod: runtime/default
Status:         Pending
IP:             
IPs:            <none>
Controlled By:  ReplicationController/macvlan-bridge-whereabouts-pod1
Containers:
  macvlan-bridge-whereabouts:
    Container ID:   
    Image:          quay.io/openshifttest/hello-sdn@sha256:2af5b5ec480f05fda7e9b278023ba04724a3dd53a296afcd8c13f220dec52197
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-qntr6 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-qntr6:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
    ConfigMapName:           openshift-service-ca.crt
    ConfigMapOptional:       <nil>
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason                  Age                  From               Message
  ----     ------                  ----                 ----               -------
  Normal   Scheduled               35m                  default-scheduler  Successfully assigned multus-upgrade/macvlan-bridge-whereabouts-pod1-5gdkz to weliang-6237-llpdj-worker-b-2kg9p.c.openshift-qe.internal by weliang-6237-llpdj-master-0.c.openshift-qe.internal
  Normal   AddedInterface          35m                  multus             Add eth0 [10.129.2.11/23] from openshift-sdn
  Warning  FailedCreatePodSandBox  35m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_macvlan-bridge-whereabouts-pod1-5gdkz_multus-upgrade_397f68b0-2ba2-4901-b228-ec1cad9a82f2_0(4bec83cfff80137f1da3c10f63f2f9d94d0f5b458195589c71b6c48b14799b91): error adding pod multus-upgrade_macvlan-bridge-whereabouts-pod1-5gdkz to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [multus-upgrade/macvlan-bridge-whereabouts-pod1-5gdkz/397f68b0-2ba2-4901-b228-ec1cad9a82f2:macvlan-bridge-whereabouts]: error adding container to network "macvlan-bridge-whereabouts": failed to create macvlan: device or resource busy
  Normal   AddedInterface          35m                  multus             Add eth0 [10.129.2.17/23] from openshift-sdn
  Warning  FailedCreatePodSandBox  35m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_macvlan-bridge-whereabouts-pod1-5gdkz_multus-upgrade_397f68b0-2ba2-4901-b228-ec1cad9a82f2_0(3b22e1b47239d52bfd6cbfb4f9ae0159a940ad036c1c90abafaaf75edb4b75db): error adding pod multus-upgrade_macvlan-bridge-whereabouts-pod1-5gdkz to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [multus-upgrade/macvlan-bridge-whereabouts-pod1-5gdkz/397f68b0-2ba2-4901-b228-ec1cad9a82f2:macvlan-bridge-whereabouts]: error adding container to network "macvlan-bridge-whereabouts": failed to create macvlan: device or resource busy
  Warning  FailedCreatePodSandBox  35m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_macvlan-bridge-whereabouts-pod1-5gdkz_multus-upgrade_397f68b0-2ba2-4901-b228-ec1cad9a82f2_0(51f95e599bf2ca808b3a7b30c6bcd9628bfd447002a1a4befbd641c35a802704): error adding pod multus-upgrade_macvlan-bridge-whereabouts-pod1-5gdkz to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [multus-upgrade/macvlan-bridge-whereabouts-pod1-5gdkz/397f68b0-2ba2-4901-b228-ec1cad9a82f2:macvlan-bridge-whereabouts]: error adding container to network "macvlan-bridge-whereabouts": failed to create macvlan: device or resource busy
  Normal   AddedInterface          35m                  multus             Add eth0 [10.129.2.18/23] from openshift-sdn
  Normal   AddedInterface          34m                  multus             Add eth0 [10.129.2.19/23] from openshift-sdn
  Warning  FailedCreatePodSandBox  34m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_macvlan-bridge-whereabouts-pod1-5gdkz_multus-upgrade_397f68b0-2ba2-4901-b228-ec1cad9a82f2_0(e77db7561da8fd76e0d2a9bdcf5a329207c92107769907b316253cd2e4b1a89a): error adding pod multus-upgrade_macvlan-bridge-whereabouts-pod1-5gdkz to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [multus-upgrade/macvlan-bridge-whereabouts-pod1-5gdkz/397f68b0-2ba2-4901-b228-ec1cad9a82f2:macvlan-bridge-whereabouts]: error adding container to network "macvlan-bridge-whereabouts": failed to create macvlan: device or resource busy
  Normal   AddedInterface          34m                  multus             Add eth0 [10.129.2.20/23] from openshift-sdn
  Warning  FailedCreatePodSandBox  34m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_macvlan-bridge-whereabouts-pod1-5gdkz_multus-upgrade_397f68b0-2ba2-4901-b228-ec1cad9a82f2_0(63babceaba7b482b2299569de05c3a93d78dbd8664c4cd714e70b0d68fb4fdf8): error adding pod multus-upgrade_macvlan-bridge-whereabouts-pod1-5gdkz to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [multus-upgrade/macvlan-bridge-whereabouts-pod1-5gdkz/397f68b0-2ba2-4901-b228-ec1cad9a82f2:macvlan-bridge-whereabouts]: error adding container to network "macvlan-bridge-whereabouts": failed to create macvlan: device or resource busy
  Normal   AddedInterface          34m                  multus             Add eth0 [10.129.2.21/23] from openshift-sdn
  Warning  FailedCreatePodSandBox  34m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_macvlan-bridge-whereabouts-pod1-5gdkz_multus-upgrade_397f68b0-2ba2-4901-b228-ec1cad9a82f2_0(8cb0cc09ef74c6c6ea344b3c93674d945ec372680a73a6e252a8d8bd46434bef): error adding pod multus-upgrade_macvlan-bridge-whereabouts-pod1-5gdkz to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [multus-upgrade/macvlan-bridge-whereabouts-pod1-5gdkz/397f68b0-2ba2-4901-b228-ec1cad9a82f2:macvlan-bridge-whereabouts]: error adding container to network "macvlan-bridge-whereabouts": failed to create macvlan: device or resource busy
  Normal   AddedInterface          34m                  multus             Add eth0 [10.129.2.22/23] from openshift-sdn
  Warning  FailedCreatePodSandBox  34m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_macvlan-bridge-whereabouts-pod1-5gdkz_multus-upgrade_397f68b0-2ba2-4901-b228-ec1cad9a82f2_0(a21229b33d72ceab98f059ac531e9a68e48d32e114666f9317ffe7a2f95d0cfd): error adding pod multus-upgrade_macvlan-bridge-whereabouts-pod1-5gdkz to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [multus-upgrade/macvlan-bridge-whereabouts-pod1-5gdkz/397f68b0-2ba2-4901-b228-ec1cad9a82f2:macvlan-bridge-whereabouts]: error adding container to network "macvlan-bridge-whereabouts": failed to create macvlan: device or resource busy
  Normal   AddedInterface          33m                  multus             Add eth0 [10.129.2.23/23] from openshift-sdn
  Warning  FailedCreatePodSandBox  33m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_macvlan-bridge-whereabouts-pod1-5gdkz_multus-upgrade_397f68b0-2ba2-4901-b228-ec1cad9a82f2_0(b57dc15035ce63bfdb9deede182c43bdfa68fc7fac4fa3fdf7e792cc8d9a3288): error adding pod multus-upgrade_macvlan-bridge-whereabouts-pod1-5gdkz to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [multus-upgrade/macvlan-bridge-whereabouts-pod1-5gdkz/397f68b0-2ba2-4901-b228-ec1cad9a82f2:macvlan-bridge-whereabouts]: error adding container to network "macvlan-bridge-whereabouts": failed to create macvlan: device or resource busy
  Normal   AddedInterface          33m                  multus             Add eth0 [10.129.2.24/23] from openshift-sdn
  Warning  FailedCreatePodSandBox  33m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_macvlan-bridge-whereabouts-pod1-5gdkz_multus-upgrade_397f68b0-2ba2-4901-b228-ec1cad9a82f2_0(da95fb4eed6290a468892195073c5a9f91cea76ce907b25a272cec83b69bec3a): error adding pod multus-upgrade_macvlan-bridge-whereabouts-pod1-5gdkz to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [multus-upgrade/macvlan-bridge-whereabouts-pod1-5gdkz/397f68b0-2ba2-4901-b228-ec1cad9a82f2:macvlan-bridge-whereabouts]: error adding container to network "macvlan-bridge-whereabouts": failed to create macvlan: device or resource busy
  Normal   AddedInterface          33m                  multus             Add eth0 [10.129.2.25/23] from openshift-sdn
  Normal   AddedInterface          33m                  multus             Add eth0 [10.129.2.26/23] from openshift-sdn
  Normal   AddedInterface          32m                  multus             Add eth0 [10.129.2.27/23] from openshift-sdn
  Normal   AddedInterface          32m                  multus             Add eth0 [10.129.2.28/23] from openshift-sdn
  Normal   AddedInterface          32m                  multus             Add eth0 [10.129.2.29/23] from openshift-sdn
  Normal   AddedInterface          32m                  multus             Add eth0 [10.129.2.30/23] from openshift-sdn
  Normal   AddedInterface          31m                  multus             Add eth0 [10.129.2.31/23] from openshift-sdn
  Normal   AddedInterface          31m                  multus             Add eth0 [10.129.2.32/23] from openshift-sdn
  Normal   AddedInterface          31m                  multus             Add eth0 [10.129.2.33/23] from openshift-sdn
  Normal   AddedInterface          31m                  multus             Add eth0 [10.129.2.34/23] from openshift-sdn
  Normal   AddedInterface          30m                  multus             Add eth0 [10.129.2.35/23] from openshift-sdn
  Normal   AddedInterface          30m                  multus             Add eth0 [10.129.2.36/23] from openshift-sdn
  Normal   AddedInterface          30m                  multus             Add eth0 [10.129.2.37/23] from openshift-sdn
  Normal   AddedInterface          30m                  multus             Add eth0 [10.129.2.38/23] from openshift-sdn
  Normal   AddedInterface          29m                  multus             Add eth0 [10.129.2.39/23] from openshift-sdn
  Warning  FailedCreatePodSandBox  29m (x15 over 33m)   kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_macvlan-bridge-whereabouts-pod1-5gdkz_multus-upgrade_397f68b0-2ba2-4901-b228-ec1cad9a82f2_0(97c4f3fd5bf2a64fcfe43bf5df1b4067ad951b34cd850adf6b75837ecabbeef9): error adding pod multus-upgrade_macvlan-bridge-whereabouts-pod1-5gdkz to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [multus-upgrade/macvlan-bridge-whereabouts-pod1-5gdkz/397f68b0-2ba2-4901-b228-ec1cad9a82f2:macvlan-bridge-whereabouts]: error adding container to network "macvlan-bridge-whereabouts": failed to create macvlan: device or resource busy
  Warning  FailedCreatePodSandBox  24m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_macvlan-bridge-whereabouts-pod1-5gdkz_multus-upgrade_397f68b0-2ba2-4901-b228-ec1cad9a82f2_0(89c2467ff0a21d0114646fd83668d09f819af1d8626cdf53fea138755cdea93b): error adding pod multus-upgrade_macvlan-bridge-whereabouts-pod1-5gdkz to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): Multus: [multus-upgrade/macvlan-bridge-whereabouts-pod1-5gdkz/397f68b0-2ba2-4901-b228-ec1cad9a82f2]: have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition
  Normal   AddedInterface          24m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Warning  FailedCreatePodSandBox  23m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_macvlan-bridge-whereabouts-pod1-5gdkz_multus-upgrade_397f68b0-2ba2-4901-b228-ec1cad9a82f2_0(68a0310f260db13079ae45c906243b4f31cee7215753a97282fae75c12636841): error adding pod multus-upgrade_macvlan-bridge-whereabouts-pod1-5gdkz to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [multus-upgrade/macvlan-bridge-whereabouts-pod1-5gdkz/397f68b0-2ba2-4901-b228-ec1cad9a82f2:macvlan-bridge-whereabouts]: error adding container to network "macvlan-bridge-whereabouts": failed to create macvlan: device or resource busy
  Normal   AddedInterface          23m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Warning  FailedCreatePodSandBox  23m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_macvlan-bridge-whereabouts-pod1-5gdkz_multus-upgrade_397f68b0-2ba2-4901-b228-ec1cad9a82f2_0(02beee5afbe13c6e1f922d145e540f4cee686ac0188e14cd7524e2a1af22b425): error adding pod multus-upgrade_macvlan-bridge-whereabouts-pod1-5gdkz to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [multus-upgrade/macvlan-bridge-whereabouts-pod1-5gdkz/397f68b0-2ba2-4901-b228-ec1cad9a82f2:macvlan-bridge-whereabouts]: error adding container to network "macvlan-bridge-whereabouts": failed to create macvlan: device or resource busy
  Warning  FailedCreatePodSandBox  23m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_macvlan-bridge-whereabouts-pod1-5gdkz_multus-upgrade_397f68b0-2ba2-4901-b228-ec1cad9a82f2_0(079c3c1f36684264a257f3f3aa11d131d8dec26407d1077bc045ad922aa8e282): error adding pod multus-upgrade_macvlan-bridge-whereabouts-pod1-5gdkz to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [multus-upgrade/macvlan-bridge-whereabouts-pod1-5gdkz/397f68b0-2ba2-4901-b228-ec1cad9a82f2:macvlan-bridge-whereabouts]: error adding container to network "macvlan-bridge-whereabouts": failed to create macvlan: device or resource busy
  Normal   AddedInterface          23m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          23m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Warning  FailedCreatePodSandBox  23m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_macvlan-bridge-whereabouts-pod1-5gdkz_multus-upgrade_397f68b0-2ba2-4901-b228-ec1cad9a82f2_0(e9dc758291a7c9998efa488e56472bd7c1ade88d62f53649f3e9c3e7204b3fbf): error adding pod multus-upgrade_macvlan-bridge-whereabouts-pod1-5gdkz to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [multus-upgrade/macvlan-bridge-whereabouts-pod1-5gdkz/397f68b0-2ba2-4901-b228-ec1cad9a82f2:macvlan-bridge-whereabouts]: error adding container to network "macvlan-bridge-whereabouts": failed to create macvlan: device or resource busy
  Normal   AddedInterface          22m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Warning  FailedCreatePodSandBox  22m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_macvlan-bridge-whereabouts-pod1-5gdkz_multus-upgrade_397f68b0-2ba2-4901-b228-ec1cad9a82f2_0(9b91db6ef8bc62d91f5235111a3cbb29a65cd2622fd224770cd631e85d96e8f8): error adding pod multus-upgrade_macvlan-bridge-whereabouts-pod1-5gdkz to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [multus-upgrade/macvlan-bridge-whereabouts-pod1-5gdkz/397f68b0-2ba2-4901-b228-ec1cad9a82f2:macvlan-bridge-whereabouts]: error adding container to network "macvlan-bridge-whereabouts": failed to create macvlan: device or resource busy
  Normal   AddedInterface          22m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Warning  FailedCreatePodSandBox  22m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_macvlan-bridge-whereabouts-pod1-5gdkz_multus-upgrade_397f68b0-2ba2-4901-b228-ec1cad9a82f2_0(2d6e8bc9d15aea9a80d1c5eb81e8414c438288ab8cf998ddd0b98c77b9b64a84): error adding pod multus-upgrade_macvlan-bridge-whereabouts-pod1-5gdkz to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [multus-upgrade/macvlan-bridge-whereabouts-pod1-5gdkz/397f68b0-2ba2-4901-b228-ec1cad9a82f2:macvlan-bridge-whereabouts]: error adding container to network "macvlan-bridge-whereabouts": failed to create macvlan: device or resource busy
  Warning  FailedCreatePodSandBox  22m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_macvlan-bridge-whereabouts-pod1-5gdkz_multus-upgrade_397f68b0-2ba2-4901-b228-ec1cad9a82f2_0(4a0ac45f27c67a4b0c7dd801bbf37ceb001f0d371f952856b3228cc27295fb1f): error adding pod multus-upgrade_macvlan-bridge-whereabouts-pod1-5gdkz to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [multus-upgrade/macvlan-bridge-whereabouts-pod1-5gdkz/397f68b0-2ba2-4901-b228-ec1cad9a82f2:macvlan-bridge-whereabouts]: error adding container to network "macvlan-bridge-whereabouts": failed to create macvlan: device or resource busy
  Normal   AddedInterface          22m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          22m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Warning  FailedCreatePodSandBox  22m                  kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_macvlan-bridge-whereabouts-pod1-5gdkz_multus-upgrade_397f68b0-2ba2-4901-b228-ec1cad9a82f2_0(bf2cf1c6a98fe2a39a2741c2ac47ee3a67366e48254760e2d64bc96f3158052d): error adding pod multus-upgrade_macvlan-bridge-whereabouts-pod1-5gdkz to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [multus-upgrade/macvlan-bridge-whereabouts-pod1-5gdkz/397f68b0-2ba2-4901-b228-ec1cad9a82f2:macvlan-bridge-whereabouts]: error adding container to network "macvlan-bridge-whereabouts": failed to create macvlan: device or resource busy
  Normal   AddedInterface          21m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          21m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          21m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          20m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          20m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          20m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          20m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          19m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          19m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          19m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          19m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          18m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          18m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          18m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          17m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          17m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          17m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          17m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          16m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          16m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          16m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          16m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          15m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          15m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          15m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          14m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          14m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          14m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          14m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          13m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          13m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          13m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          13m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          12m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          12m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          11m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          11m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          11m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          11m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          10m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          10m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          10m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          10m                  multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          9m49s                multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          9m31s                multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          9m18s                multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          8m59s                multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          8m45s                multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          8m30s                multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          8m15s                multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          7m58s                multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          7m42s                multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          7m24s                multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          7m9s                 multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          6m53s                multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          6m37s                multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          6m25s                multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          6m7s                 multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          5m49s                multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          5m34s                multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          5m18s                multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          5m3s                 multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          4m46s                multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          4m32s                multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          4m18s                multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Normal   AddedInterface          4m4s                 multus             Add eth0 [10.130.0.16/23] from ovn-kubernetes
  Warning  FailedCreatePodSandBox  4m3s (x66 over 21m)  kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_macvlan-bridge-whereabouts-pod1-5gdkz_multus-upgrade_397f68b0-2ba2-4901-b228-ec1cad9a82f2_0(98b458a89db3a00a3d2b28b9642826900c9a58528398588e7fb9e6522fe5069d): error adding pod multus-upgrade_macvlan-bridge-whereabouts-pod1-5gdkz to CNI network "multus-cni-network": plugin type="multus" name="multus-cni-network" failed (add): [multus-upgrade/macvlan-bridge-whereabouts-pod1-5gdkz/397f68b0-2ba2-4901-b228-ec1cad9a82f2:macvlan-bridge-whereabouts]: error adding container to network "macvlan-bridge-whereabouts": failed to create macvlan: device or resource busy


Actual results:
Multus pod got stuck at ContainerCreating state after SDN to OVN migration.

Expected results:
Multus pod should not get stuck at ContainerCreating state after SDN to OVN migration.


Additional info:

Tested in GCP

Comment 1 Douglas Smith 2022-06-24 13:59:53 UTC
Couple things in the pod events that stand out here to me are:

> error adding container to network "macvlan-bridge-whereabouts": failed to create macvlan: device or resource busy

There might be something going on with the interface on the host?

Nikhil, also note that upstream you had filed: https://github.com/k8snetworkplumbingwg/multus-cni/issues/715 (which went stale)

> have you checked that your default network is ready? still waiting for readinessindicatorfile @ /var/run/multus/cni/net.d/10-ovn-kubernetes.conf. pollimmediate error: timed out waiting for the condition

Looks like it still may be in the process of OVN-K launching, too.

Comment 2 Weibin Liang 2022-07-07 17:55:11 UTC
Bug may be caused https://bugzilla.redhat.com/show_bug.cgi?id=2100643

Retest with https://github.com/openshift/release/pull/29821, the test passed in 4.11.0-rc.0

[weliang@weliang openshift-tests-private]$ oc new-project multus-upgrade
[weliang@weliang multus]$ oc create -f Multus-NAD1.yaml
networkattachmentdefinition.k8s.cni.cncf.io/macvlan-nad1 created
[weliang@weliang multus]$ oc create -f Multus-pod-migration.yaml
replicationcontroller/test-pod created
[weliang@weliang multus]$ 
[weliang@weliang openshift-tests-private]$ oc get pod
NAME             READY   STATUS    RESTARTS   AGE
test-pod-jmhfz   1/1     Running   0          50s
test-pod-ndw89   1/1     Running   0          50s
test-pod-qnnwj   1/1     Running   0          50s
[weliang@weliang openshift-tests-private]$ 

./ovn-sdn-migration-commands.sh
[weliang@weliang openshift-tests-private]$ oc get pod
NAME             READY   STATUS    RESTARTS   AGE
test-pod-dhppk   1/1     Running   1          16m
test-pod-g5nnh   1/1     Running   1          19m
test-pod-tbcnm   1/1     Running   1          16m

./ovn-sdn-migration-rollback-commands.sh
[weliang@weliang openshift-tests-private]$ oc get pod
NAME             READY   STATUS    RESTARTS   AGE
test-pod-69zsl   1/1     Running   0          50m
test-pod-dq45w   1/1     Running   0          50m
test-pod-rxv2l   1/1     Running   0          47m
[weliang@weliang openshift-tests-private]$ oc get clusterversion
NAME      VERSION       AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.0-rc.0   True        False         3h21m   Cluster version is 4.11.0-rc.0
[weliang@weliang openshift-tests-private]$

*** This bug has been marked as a duplicate of bug 2100643 ***