Bug 1656916 - Multus err in tearing down failed plugins with successful creation of VMI
Summary: Multus err in tearing down failed plugins with successful creation of VMI
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Networking
Version: 1.3
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: ---
Assignee: Dan Kenigsberg
QA Contact: Meni Yakove
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-12-06 16:18 UTC by Aditya Ramteke
Modified: 2018-12-11 13:08 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-12-11 13:08:17 UTC
Target Upstream Version:


Attachments (Terms of Use)

Description Aditya Ramteke 2018-12-06 16:18:45 UTC
Description of problem:

➜  ~ oc describe pod virt-launcher-vmi-multus-ptp-5kfch
Name:               virt-launcher-vmi-multus-ptp-5kfch
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               cnv-executor-aditya-node2.example.com/172.16.0.25
Start Time:         Thu, 06 Dec 2018 20:46:26 +0530
Labels:             kubevirt.io=virt-launcher
                    kubevirt.io/created-by=e63e4aba-f969-11e8-8ef1-fa163e890c9f
                    kubevirt.io/vm=vm-fedora
Annotations:        k8s.v1.cni.cncf.io/networks=sample-net-attach-def
                    k8s.v1.cni.cncf.io/networks-status=
                    kubevirt.io/domain=vmi-multus-ptp
                    kubevirt.io/owned-by=virt-controller
                    openshift.io/scc=privileged
Status:             Pending
IP:                 
Containers:
  compute:
    Container ID:  
    Image:         brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cnv-tech-preview/virt-launcher:v1.3.0
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/virt-launcher
      --qemu-timeout
      5m
      --name
      vmi-multus-ptp
      --uid
      e63e4aba-f969-11e8-8ef1-fa163e890c9f
      --namespace
      default
      --kubevirt-share-dir
      /var/run/kubevirt
      --ephemeral-disk-dir
      /var/run/kubevirt-ephemeral-disks
      --readiness-file
      /tmp/healthy
      --grace-period-seconds
      45
      --hook-sidecars
      0
      --less-pvc-space-toleration
      10
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      devices.kubevirt.io/kvm:        1
      devices.kubevirt.io/tun:        1
      devices.kubevirt.io/vhost-net:  1
    Requests:
      devices.kubevirt.io/kvm:        1
      devices.kubevirt.io/tun:        1
      devices.kubevirt.io/vhost-net:  1
      memory:                         103476Ki
    Readiness:                        exec [cat /tmp/healthy] delay=2s timeout=5s period=2s #success=1 #failure=5
    Environment:                      <none>
    Mounts:
      /var/run/kubevirt from virt-share-dir (rw)
      /var/run/kubevirt-ephemeral-disks from ephemeral-disks (rw)
      /var/run/libvirt from libvirt-runtime (rw)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  virt-share-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/kubevirt
    HostPathType:  
  libvirt-runtime:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  
  ephemeral-disks:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:      
QoS Class:       Burstable
Node-Selectors:  kubevirt.io/schedulable=true
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule
Events:
  Type     Reason                  Age                From                                            Message
  ----     ------                  ----               ----                                            -------
  Normal   Scheduled               30m                default-scheduler                               Successfully assigned default/virt-launcher-vmi-multus-ptp-5kfch to cnv-executor-aditya-node2.example.com
  Warning  FailedCreatePodSandBox  29m                kubelet, cnv-executor-aditya-node2.example.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-5kfch_default_e640f6c6-f969-11e8-8ef1-fa163e890c9f_0(9b7d49d0ecfb8ae24f11ef0a9100929db0573eb8240ad208a59b072d6ea0a9b6): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found
  Warning  FailedCreatePodSandBox  29m                kubelet, cnv-executor-aditya-node2.example.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-5kfch_default_e640f6c6-f969-11e8-8ef1-fa163e890c9f_0(2e3fcacab69a20d655b46f868adcddfbf9cc169cdf85f5b453b69d3adb7ac75d): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found
  Warning  FailedCreatePodSandBox  29m                kubelet, cnv-executor-aditya-node2.example.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-5kfch_default_e640f6c6-f969-11e8-8ef1-fa163e890c9f_0(900ace6a502c5bca6ad6d523381c85d6c6650f59508562cf09b7a67c54f5b31d): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found
  Warning  FailedCreatePodSandBox  29m                kubelet, cnv-executor-aditya-node2.example.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-5kfch_default_e640f6c6-f969-11e8-8ef1-fa163e890c9f_0(190d45d86012b7a4b7a4021dce8b1690cc54c78bf3e4259a7c6e8bccf92684be): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found
  Warning  FailedCreatePodSandBox  29m                kubelet, cnv-executor-aditya-node2.example.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-5kfch_default_e640f6c6-f969-11e8-8ef1-fa163e890c9f_0(7a697eb2c76639ee67128b1b109cdf079804eb18b912fc86ad1f7607ac5e9a8e): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found
  Warning  FailedCreatePodSandBox  28m                kubelet, cnv-executor-aditya-node2.example.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-5kfch_default_e640f6c6-f969-11e8-8ef1-fa163e890c9f_0(cdcfe56f8cff060f675a3f36368e5dec17cb04c837f568bed74b8dad40e594a7): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found
  Warning  FailedCreatePodSandBox  28m                kubelet, cnv-executor-aditya-node2.example.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-5kfch_default_e640f6c6-f969-11e8-8ef1-fa163e890c9f_0(c70ddf1dc7be025b891a5b0e4d53de7901963c0aafccf0713e69fc761b6070e7): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found
  Warning  FailedCreatePodSandBox  28m                kubelet, cnv-executor-aditya-node2.example.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-5kfch_default_e640f6c6-f969-11e8-8ef1-fa163e890c9f_0(125642d3cfcbfb7a18ed6c7815a4dd587e94a500ce2eea4a8be0e282ec1b0d04): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found
  Warning  FailedCreatePodSandBox  28m                kubelet, cnv-executor-aditya-node2.example.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-5kfch_default_e640f6c6-f969-11e8-8ef1-fa163e890c9f_0(c7fb9534afb817e90e363881e666f5d2ae7459b5d880cd0f57e37bb23fd4467e): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found
  Warning  FailedCreatePodSandBox  4m (x90 over 27m)  kubelet, cnv-executor-aditya-node2.example.com  (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-5kfch_default_e640f6c6-f969-11e8-8ef1-fa163e890c9f_0(08af10a9f79f5aa4d03e2b5b063def86d04de36316e6b92f4769d61efa5e11aa): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Create NetworkAttachmentDefinition
2. Create VM
3. Start VM

Actual results:
VM started successfully and status of a virt-launcher container is `ContainerCreating` having events `Warning  FailedCreatePodSandBox`

Expected results:
VM is started and in running.

Additional info:

➜ cat network-attachment.yaml 
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: sample-net-attach-def
spec:
  config: '{
      "cniVersion": "0.3.1",
      "type": "ovs",
      "bridge": "br1",
      "vlan": 100
    }'

➜ oc create -f network-attachment.yaml


➜ oc describe net-attach-def sample-net-attach-def                     
Name:         sample-net-attach-def
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  k8s.cni.cncf.io/v1
Kind:         NetworkAttachmentDefinition
Metadata:
  Creation Timestamp:  2018-12-06T15:00:03Z
  Generation:          1
  Resource Version:    32812
  Self Link:           /apis/k8s.cni.cncf.io/v1/namespaces/default/network-attachment-definitions/sample-net-attach-def
  UID:                 9c75003b-f967-11e8-8ef1-fa163e890c9f
Spec:
  Config:  { "cniVersion": "0.3.1", "type": "ovs", "bridge": "br1", "vlan": 100 }
Events:    <none>


➜ cat vmi-multus-ptp.yaml 
apiVersion: kubevirt.io/v1alpha2
kind: VirtualMachine
metadata:
  creationTimestamp: null
  labels:
    special: vmi-multus-ptp
  name: vmi-multus-ptp
spec:
  running: false
  template:
    metadata:
      creationTimestamp: null
      labels:
        kubevirt.io/vm: vm-fedora
    spec:
      domain:
        devices:
          interfaces:
            - name: default
              bridge: {}
            - name: ovs-net
              bridge: {}
      networks:
      - name: default
        pod: {} # Stock pod network
      - name: ovs-net
        multus: # Secondary multus network
          networkName: sample-net-attach-def
      disks:
          - name: registrydisk
            volume:
              registryDisk:
                image: kubevirt/fedora-cloud-registry-disk-demo:latest
            disk:
              bus: virtio
status: {}

➜ oc create -f vmi-multus-ptp.yaml

➜ oc get vm
vmi-multus-ptp   58m

➜ virtctl start vmi-multus-ptp
VM vmi-multus-ptp was scheduled to start


➜ oc describe vmi vmi-multus-ptp
Name:         vmi-multus-ptp
Namespace:    default
Labels:       kubevirt.io/vm=vm-fedora
Annotations:  <none>
API Version:  kubevirt.io/v1alpha2
Kind:         VirtualMachineInstance
Metadata:
  Creation Timestamp:  2018-12-06T16:15:55Z
  Finalizers:
    foregroundDeleteVirtualMachine
  Generate Name:  vmi-multus-ptp
  Generation:     1
  Owner References:
    API Version:           kubevirt.io/v1alpha2
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  VirtualMachine
    Name:                  vmi-multus-ptp
    UID:                   c50ccb6a-f969-11e8-8ef1-fa163e890c9f
  Resource Version:        45068
  Self Link:               /apis/kubevirt.io/v1alpha2/namespaces/default/virtualmachineinstances/vmi-multus-ptp
  UID:                     358b4dc6-f972-11e8-8ef1-fa163e890c9f
Spec:
  Domain:
    Devices:
      Interfaces:
        Bridge:
        Name:  default
        Bridge:
        Name:  ovs-net
    Features:
      Acpi:
        Enabled:  true
    Firmware:
      Uuid:  d593504e-2fe0-5555-b35c-41e88af44956
    Machine:
      Type:  q35
    Resources:
      Requests:
        Memory:  8Mi
  Networks:
    Name:  default
    Pod:
    Multus:
      Network Name:  sample-net-attach-def
    Name:            ovs-net
Status:
  Phase:  Scheduling
Events:
  Type    Reason            Age   From                       Message
  ----    ------            ----  ----                       -------
  Normal  SuccessfulCreate  38s   virtualmachine-controller  Created virtual machine pod virt-launcher-vmi-multus-ptp-fb7lj

➜ oc get pods
virt-launcher-vmi-multus-ptp-fb7lj   0/1       ContainerCreating   0          1m


➜ oc describe pod virt-launcher-vmi-multus-ptp-fb7lj
Name:               virt-launcher-vmi-multus-ptp-fb7lj
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               cnv-executor-aditya-node2.example.com/172.16.0.25
Start Time:         Thu, 06 Dec 2018 21:45:55 +0530
Labels:             kubevirt.io=virt-launcher
                    kubevirt.io/created-by=358b4dc6-f972-11e8-8ef1-fa163e890c9f
                    kubevirt.io/vm=vm-fedora
Annotations:        k8s.v1.cni.cncf.io/networks=sample-net-attach-def
                    k8s.v1.cni.cncf.io/networks-status=
                    kubevirt.io/domain=vmi-multus-ptp
                    kubevirt.io/owned-by=virt-controller
                    openshift.io/scc=privileged
Status:             Pending
IP:                 
Containers:
  compute:
    Container ID:  
    Image:         brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cnv-tech-preview/virt-launcher:v1.3.0
    Image ID:      
    Port:          <none>
    Host Port:     <none>
    Command:
      /usr/bin/virt-launcher
      --qemu-timeout
      5m
      --name
      vmi-multus-ptp
      --uid
      358b4dc6-f972-11e8-8ef1-fa163e890c9f
      --namespace
      default
      --kubevirt-share-dir
      /var/run/kubevirt
      --ephemeral-disk-dir
      /var/run/kubevirt-ephemeral-disks
      --readiness-file
      /tmp/healthy
      --grace-period-seconds
      45
      --hook-sidecars
      0
      --less-pvc-space-toleration
      10
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Limits:
      devices.kubevirt.io/kvm:        1
      devices.kubevirt.io/tun:        1
      devices.kubevirt.io/vhost-net:  1
    Requests:
      devices.kubevirt.io/kvm:        1
      devices.kubevirt.io/tun:        1
      devices.kubevirt.io/vhost-net:  1
      memory:                         103476Ki
    Readiness:                        exec [cat /tmp/healthy] delay=2s timeout=5s period=2s #success=1 #failure=5
    Environment:                      <none>
    Mounts:
      /var/run/kubevirt from virt-share-dir (rw)
      /var/run/kubevirt-ephemeral-disks from ephemeral-disks (rw)
      /var/run/libvirt from libvirt-runtime (rw)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  virt-share-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/run/kubevirt
    HostPathType:  
  libvirt-runtime:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:  
  ephemeral-disks:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:      
QoS Class:       Burstable
Node-Selectors:  kubevirt.io/schedulable=true
Tolerations:     node.kubernetes.io/memory-pressure:NoSchedule
Events:
  Type     Reason                  Age   From                                            Message
  ----     ------                  ----  ----                                            -------
  Normal   Scheduled               1m    default-scheduler                               Successfully assigned default/virt-launcher-vmi-multus-ptp-fb7lj to cnv-executor-aditya-node2.example.com
  Warning  FailedCreatePodSandBox  1m    kubelet, cnv-executor-aditya-node2.example.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-fb7lj_default_358de615-f972-11e8-8ef1-fa163e890c9f_0(1a0d0523ad551fa28f0a05cc3bb946c7170bad5cd4260423b5d7e19385c6a68c): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found
  Warning  FailedCreatePodSandBox  1m    kubelet, cnv-executor-aditya-node2.example.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-fb7lj_default_358de615-f972-11e8-8ef1-fa163e890c9f_0(e93aa90f24f636bbfceb454b2be1a96cf89333d744e75b9dc5873c0e9d6371f8): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found
  Warning  FailedCreatePodSandBox  1m    kubelet, cnv-executor-aditya-node2.example.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-fb7lj_default_358de615-f972-11e8-8ef1-fa163e890c9f_0(5ed7c99e6ab18cb41ca392295440790b625191fb1b2943684bdb162c0008cb73): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found
  Warning  FailedCreatePodSandBox  1m    kubelet, cnv-executor-aditya-node2.example.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-fb7lj_default_358de615-f972-11e8-8ef1-fa163e890c9f_0(178f7be80bc842025f565d7d14e8f3987599a2ff622110efce8a7cda2eafffa0): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found
  Warning  FailedCreatePodSandBox  1m    kubelet, cnv-executor-aditya-node2.example.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-fb7lj_default_358de615-f972-11e8-8ef1-fa163e890c9f_0(c60e5cbeb262a48fdec13ed95089cc85d33f833a869a957a69d10b5cd9320a82): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found
  Warning  FailedCreatePodSandBox  58s   kubelet, cnv-executor-aditya-node2.example.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-fb7lj_default_358de615-f972-11e8-8ef1-fa163e890c9f_0(6d23fffeaff9bb4e3492a1a71a52538f99cf3377b1526332a3c70ee82bb3989f): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found
  Warning  FailedCreatePodSandBox  40s   kubelet, cnv-executor-aditya-node2.example.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-fb7lj_default_358de615-f972-11e8-8ef1-fa163e890c9f_0(982ead9eda6b5edd5fe6a4b0253ae4ee0ca7939c1f246c48863e6291346459fe): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found
  Warning  FailedCreatePodSandBox  24s   kubelet, cnv-executor-aditya-node2.example.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-fb7lj_default_358de615-f972-11e8-8ef1-fa163e890c9f_0(6c126a5421b124b785dc3509f3c8d041f478d8b7faac025004d4d2654f7d7351): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found
  Warning  FailedCreatePodSandBox  11s   kubelet, cnv-executor-aditya-node2.example.com  Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-fb7lj_default_358de615-f972-11e8-8ef1-fa163e890c9f_0(a4b0f2dd7f7e09ef76b2aade6d2014726c60b7de817bd88279fae419685c6b2f): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found

Comment 1 Yan Du 2018-12-07 05:14:04 UTC
@Aditya Please make sure you have created br1 on your nodes.

Comment 2 Aditya Ramteke 2018-12-07 09:30:30 UTC
@Yan I have installed the ovs-cni manifest rpm package and deploy the yaml files

and got some failures: 

[cloud-user@cnv-executor-aditya-master1 ~]$ sudo yum install ovs-cni-manifests
...
...
Installed:
  ovs-cni-manifests.noarch 0:0.2.0-2                                                                                                                                                                                 

Complete!


[cloud-user@cnv-executor-aditya-master1 ~]$ oc apply -f /usr/share/ovs-cni/manifests/
customresourcedefinition.apiextensions.k8s.io/network-attachment-definitions.k8s.cni.cncf.io configured
clusterrole.rbac.authorization.k8s.io/multus configured
clusterrolebinding.rbac.authorization.k8s.io/multus configured
serviceaccount/multus unchanged
configmap/multus-cni-config configured
daemonset.extensions/kube-multus-ds-amd64 created
daemonset.extensions/ovs-cni-plugin-amd64 created
The DaemonSet "ovs-vsctl-amd64" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"app":"ovs-cni", "tier":"node"}: `selector` does not match template `labels`

[cloud-user@cnv-executor-aditya-master1 ~]$ oc get pods -n kube-system 
NAME                                                         READY     STATUS             RESTARTS   AGE
cdi-api-556469bfbc-4xchf                                     1/1       Running            0          2h
cdi-deployment-6c49474554-dcr9r                              1/1       Running            0          2h
cdi-uploadproxy-784878c6cc-tl2b9                             1/1       Running            0          2h
kube-multus-amd64-fwczp                                      1/1       Running            0          2h
kube-multus-amd64-ldzck                                      1/1       Running            0          2h
kube-multus-ds-amd64-j6hg9                                   0/1       ImagePullBackOff   0          23s
kube-multus-ds-amd64-rbxqt                                   0/1       ImagePullBackOff   0          23s
kube-ovs-cni-plugin-amd64-54zlv                              1/1       Running            0          2h
kube-ovs-cni-plugin-amd64-w7kxx                              1/1       Running            0          2h
master-api-cnv-executor-aditya-master1.example.com           1/1       Running            0          3h
master-controllers-cnv-executor-aditya-master1.example.com   1/1       Running            0          3h
master-etcd-cnv-executor-aditya-master1.example.com          1/1       Running            0          3h
ovs-cni-plugin-amd64-4g54t                                   0/1       ImagePullBackOff   0          23s
ovs-cni-plugin-amd64-fc6wn                                   0/1       ImagePullBackOff   0          23s
ovs-vsctl-amd64-4hrz5                                        1/1       Running            0          2h
ovs-vsctl-amd64-jvn7v                                        1/1       Running            0          2h
virt-api-596b5678c4-tt49h                                    1/1       Running            0          2h
virt-api-596b5678c4-zlxxt                                    1/1       Running            0          2h
virt-controller-7f77fb7b95-zbrh5                             1/1       Running            0          2h
virt-controller-7f77fb7b95-zzvmt                             1/1       Running            0          2h
virt-handler-564rf                                           1/1       Running            0          2h
virt-handler-bvpmt                                           1/1       Running            0          2h


[cloud-user@cnv-executor-aditya-master1 ~]$ oc describe pod kube-multus-ds-amd64-rbxqt
....
....
Events:
  Type     Reason   Age               From                                            Message
  ----     ------   ----              ----                                            -------
  Normal   Pulling  16s (x4 over 1m)  kubelet, cnv-executor-aditya-node1.example.com  pulling image "registry.access.redhat.com/cnv13-tech-preview/multus-cni:1.3"
  Warning  Failed   15s (x4 over 1m)  kubelet, cnv-executor-aditya-node1.example.com  Failed to pull image "registry.access.redhat.com/cnv13-tech-preview/multus-cni:1.3": rpc error: code = Unknown desc = Error reading manifest 1.3 in registry.access.redhat.com/cnv13-tech-preview/multus-cni: unknown: Not Found
  Warning  Failed   15s (x4 over 1m)  kubelet, cnv-executor-aditya-node1.example.com  Error: ErrImagePull
  Normal   BackOff  4s (x6 over 1m)   kubelet, cnv-executor-aditya-node1.example.com  Back-off pulling image "registry.access.redhat.com/cnv13-tech-preview/multus-cni:1.3"
  Warning  Failed   4s (x6 over 1m)   kubelet, cnv-executor-aditya-node1.example.com  Error: ImagePullBackOff

Comment 3 Yan Du 2018-12-10 02:41:31 UTC
@Aditya I don't think the registry.access.redhat.com/cnv13-tech-preview/multus-cni:1.3 here is correct.

# docker pull registry.access.redhat.com/cnv13-tech-preview/multus-cni:1.3
Trying to pull repository registry.access.redhat.com/cnv13-tech-preview/multus-cni ... 
unknown: Not Found

Comment 4 Nelly Credi 2018-12-10 08:49:07 UTC
right, it should be registry.access.redhat.com/cnv-tech-preview

Comment 5 Dan Kenigsberg 2018-12-11 13:08:17 UTC
Please reopen if using the correct product designation does not solve.


Note You need to log in before you can comment on or make changes to this bug.