Description of problem: ➜ ~ oc describe pod virt-launcher-vmi-multus-ptp-5kfch Name: virt-launcher-vmi-multus-ptp-5kfch Namespace: default Priority: 0 PriorityClassName: <none> Node: cnv-executor-aditya-node2.example.com/172.16.0.25 Start Time: Thu, 06 Dec 2018 20:46:26 +0530 Labels: kubevirt.io=virt-launcher kubevirt.io/created-by=e63e4aba-f969-11e8-8ef1-fa163e890c9f kubevirt.io/vm=vm-fedora Annotations: k8s.v1.cni.cncf.io/networks=sample-net-attach-def k8s.v1.cni.cncf.io/networks-status= kubevirt.io/domain=vmi-multus-ptp kubevirt.io/owned-by=virt-controller openshift.io/scc=privileged Status: Pending IP: Containers: compute: Container ID: Image: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cnv-tech-preview/virt-launcher:v1.3.0 Image ID: Port: <none> Host Port: <none> Command: /usr/bin/virt-launcher --qemu-timeout 5m --name vmi-multus-ptp --uid e63e4aba-f969-11e8-8ef1-fa163e890c9f --namespace default --kubevirt-share-dir /var/run/kubevirt --ephemeral-disk-dir /var/run/kubevirt-ephemeral-disks --readiness-file /tmp/healthy --grace-period-seconds 45 --hook-sidecars 0 --less-pvc-space-toleration 10 State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Limits: devices.kubevirt.io/kvm: 1 devices.kubevirt.io/tun: 1 devices.kubevirt.io/vhost-net: 1 Requests: devices.kubevirt.io/kvm: 1 devices.kubevirt.io/tun: 1 devices.kubevirt.io/vhost-net: 1 memory: 103476Ki Readiness: exec [cat /tmp/healthy] delay=2s timeout=5s period=2s #success=1 #failure=5 Environment: <none> Mounts: /var/run/kubevirt from virt-share-dir (rw) /var/run/kubevirt-ephemeral-disks from ephemeral-disks (rw) /var/run/libvirt from libvirt-runtime (rw) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: virt-share-dir: Type: HostPath (bare host directory volume) Path: /var/run/kubevirt HostPathType: libvirt-runtime: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: ephemeral-disks: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: QoS Class: Burstable Node-Selectors: kubevirt.io/schedulable=true Tolerations: node.kubernetes.io/memory-pressure:NoSchedule Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 30m default-scheduler Successfully assigned default/virt-launcher-vmi-multus-ptp-5kfch to cnv-executor-aditya-node2.example.com Warning FailedCreatePodSandBox 29m kubelet, cnv-executor-aditya-node2.example.com Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-5kfch_default_e640f6c6-f969-11e8-8ef1-fa163e890c9f_0(9b7d49d0ecfb8ae24f11ef0a9100929db0573eb8240ad208a59b072d6ea0a9b6): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found Warning FailedCreatePodSandBox 29m kubelet, cnv-executor-aditya-node2.example.com Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-5kfch_default_e640f6c6-f969-11e8-8ef1-fa163e890c9f_0(2e3fcacab69a20d655b46f868adcddfbf9cc169cdf85f5b453b69d3adb7ac75d): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found Warning FailedCreatePodSandBox 29m kubelet, cnv-executor-aditya-node2.example.com Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-5kfch_default_e640f6c6-f969-11e8-8ef1-fa163e890c9f_0(900ace6a502c5bca6ad6d523381c85d6c6650f59508562cf09b7a67c54f5b31d): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found Warning FailedCreatePodSandBox 29m kubelet, cnv-executor-aditya-node2.example.com Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-5kfch_default_e640f6c6-f969-11e8-8ef1-fa163e890c9f_0(190d45d86012b7a4b7a4021dce8b1690cc54c78bf3e4259a7c6e8bccf92684be): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found Warning FailedCreatePodSandBox 29m kubelet, cnv-executor-aditya-node2.example.com Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-5kfch_default_e640f6c6-f969-11e8-8ef1-fa163e890c9f_0(7a697eb2c76639ee67128b1b109cdf079804eb18b912fc86ad1f7607ac5e9a8e): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found Warning FailedCreatePodSandBox 28m kubelet, cnv-executor-aditya-node2.example.com Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-5kfch_default_e640f6c6-f969-11e8-8ef1-fa163e890c9f_0(cdcfe56f8cff060f675a3f36368e5dec17cb04c837f568bed74b8dad40e594a7): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found Warning FailedCreatePodSandBox 28m kubelet, cnv-executor-aditya-node2.example.com Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-5kfch_default_e640f6c6-f969-11e8-8ef1-fa163e890c9f_0(c70ddf1dc7be025b891a5b0e4d53de7901963c0aafccf0713e69fc761b6070e7): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found Warning FailedCreatePodSandBox 28m kubelet, cnv-executor-aditya-node2.example.com Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-5kfch_default_e640f6c6-f969-11e8-8ef1-fa163e890c9f_0(125642d3cfcbfb7a18ed6c7815a4dd587e94a500ce2eea4a8be0e282ec1b0d04): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found Warning FailedCreatePodSandBox 28m kubelet, cnv-executor-aditya-node2.example.com Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-5kfch_default_e640f6c6-f969-11e8-8ef1-fa163e890c9f_0(c7fb9534afb817e90e363881e666f5d2ae7459b5d880cd0f57e37bb23fd4467e): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found Warning FailedCreatePodSandBox 4m (x90 over 27m) kubelet, cnv-executor-aditya-node2.example.com (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-5kfch_default_e640f6c6-f969-11e8-8ef1-fa163e890c9f_0(08af10a9f79f5aa4d03e2b5b063def86d04de36316e6b92f4769d61efa5e11aa): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. Create NetworkAttachmentDefinition 2. Create VM 3. Start VM Actual results: VM started successfully and status of a virt-launcher container is `ContainerCreating` having events `Warning FailedCreatePodSandBox` Expected results: VM is started and in running. Additional info: ➜ cat network-attachment.yaml apiVersion: "k8s.cni.cncf.io/v1" kind: NetworkAttachmentDefinition metadata: name: sample-net-attach-def spec: config: '{ "cniVersion": "0.3.1", "type": "ovs", "bridge": "br1", "vlan": 100 }' ➜ oc create -f network-attachment.yaml ➜ oc describe net-attach-def sample-net-attach-def Name: sample-net-attach-def Namespace: default Labels: <none> Annotations: <none> API Version: k8s.cni.cncf.io/v1 Kind: NetworkAttachmentDefinition Metadata: Creation Timestamp: 2018-12-06T15:00:03Z Generation: 1 Resource Version: 32812 Self Link: /apis/k8s.cni.cncf.io/v1/namespaces/default/network-attachment-definitions/sample-net-attach-def UID: 9c75003b-f967-11e8-8ef1-fa163e890c9f Spec: Config: { "cniVersion": "0.3.1", "type": "ovs", "bridge": "br1", "vlan": 100 } Events: <none> ➜ cat vmi-multus-ptp.yaml apiVersion: kubevirt.io/v1alpha2 kind: VirtualMachine metadata: creationTimestamp: null labels: special: vmi-multus-ptp name: vmi-multus-ptp spec: running: false template: metadata: creationTimestamp: null labels: kubevirt.io/vm: vm-fedora spec: domain: devices: interfaces: - name: default bridge: {} - name: ovs-net bridge: {} networks: - name: default pod: {} # Stock pod network - name: ovs-net multus: # Secondary multus network networkName: sample-net-attach-def disks: - name: registrydisk volume: registryDisk: image: kubevirt/fedora-cloud-registry-disk-demo:latest disk: bus: virtio status: {} ➜ oc create -f vmi-multus-ptp.yaml ➜ oc get vm vmi-multus-ptp 58m ➜ virtctl start vmi-multus-ptp VM vmi-multus-ptp was scheduled to start ➜ oc describe vmi vmi-multus-ptp Name: vmi-multus-ptp Namespace: default Labels: kubevirt.io/vm=vm-fedora Annotations: <none> API Version: kubevirt.io/v1alpha2 Kind: VirtualMachineInstance Metadata: Creation Timestamp: 2018-12-06T16:15:55Z Finalizers: foregroundDeleteVirtualMachine Generate Name: vmi-multus-ptp Generation: 1 Owner References: API Version: kubevirt.io/v1alpha2 Block Owner Deletion: true Controller: true Kind: VirtualMachine Name: vmi-multus-ptp UID: c50ccb6a-f969-11e8-8ef1-fa163e890c9f Resource Version: 45068 Self Link: /apis/kubevirt.io/v1alpha2/namespaces/default/virtualmachineinstances/vmi-multus-ptp UID: 358b4dc6-f972-11e8-8ef1-fa163e890c9f Spec: Domain: Devices: Interfaces: Bridge: Name: default Bridge: Name: ovs-net Features: Acpi: Enabled: true Firmware: Uuid: d593504e-2fe0-5555-b35c-41e88af44956 Machine: Type: q35 Resources: Requests: Memory: 8Mi Networks: Name: default Pod: Multus: Network Name: sample-net-attach-def Name: ovs-net Status: Phase: Scheduling Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulCreate 38s virtualmachine-controller Created virtual machine pod virt-launcher-vmi-multus-ptp-fb7lj ➜ oc get pods virt-launcher-vmi-multus-ptp-fb7lj 0/1 ContainerCreating 0 1m ➜ oc describe pod virt-launcher-vmi-multus-ptp-fb7lj Name: virt-launcher-vmi-multus-ptp-fb7lj Namespace: default Priority: 0 PriorityClassName: <none> Node: cnv-executor-aditya-node2.example.com/172.16.0.25 Start Time: Thu, 06 Dec 2018 21:45:55 +0530 Labels: kubevirt.io=virt-launcher kubevirt.io/created-by=358b4dc6-f972-11e8-8ef1-fa163e890c9f kubevirt.io/vm=vm-fedora Annotations: k8s.v1.cni.cncf.io/networks=sample-net-attach-def k8s.v1.cni.cncf.io/networks-status= kubevirt.io/domain=vmi-multus-ptp kubevirt.io/owned-by=virt-controller openshift.io/scc=privileged Status: Pending IP: Containers: compute: Container ID: Image: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cnv-tech-preview/virt-launcher:v1.3.0 Image ID: Port: <none> Host Port: <none> Command: /usr/bin/virt-launcher --qemu-timeout 5m --name vmi-multus-ptp --uid 358b4dc6-f972-11e8-8ef1-fa163e890c9f --namespace default --kubevirt-share-dir /var/run/kubevirt --ephemeral-disk-dir /var/run/kubevirt-ephemeral-disks --readiness-file /tmp/healthy --grace-period-seconds 45 --hook-sidecars 0 --less-pvc-space-toleration 10 State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Limits: devices.kubevirt.io/kvm: 1 devices.kubevirt.io/tun: 1 devices.kubevirt.io/vhost-net: 1 Requests: devices.kubevirt.io/kvm: 1 devices.kubevirt.io/tun: 1 devices.kubevirt.io/vhost-net: 1 memory: 103476Ki Readiness: exec [cat /tmp/healthy] delay=2s timeout=5s period=2s #success=1 #failure=5 Environment: <none> Mounts: /var/run/kubevirt from virt-share-dir (rw) /var/run/kubevirt-ephemeral-disks from ephemeral-disks (rw) /var/run/libvirt from libvirt-runtime (rw) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: virt-share-dir: Type: HostPath (bare host directory volume) Path: /var/run/kubevirt HostPathType: libvirt-runtime: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: ephemeral-disks: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: QoS Class: Burstable Node-Selectors: kubevirt.io/schedulable=true Tolerations: node.kubernetes.io/memory-pressure:NoSchedule Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 1m default-scheduler Successfully assigned default/virt-launcher-vmi-multus-ptp-fb7lj to cnv-executor-aditya-node2.example.com Warning FailedCreatePodSandBox 1m kubelet, cnv-executor-aditya-node2.example.com Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-fb7lj_default_358de615-f972-11e8-8ef1-fa163e890c9f_0(1a0d0523ad551fa28f0a05cc3bb946c7170bad5cd4260423b5d7e19385c6a68c): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found Warning FailedCreatePodSandBox 1m kubelet, cnv-executor-aditya-node2.example.com Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-fb7lj_default_358de615-f972-11e8-8ef1-fa163e890c9f_0(e93aa90f24f636bbfceb454b2be1a96cf89333d744e75b9dc5873c0e9d6371f8): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found Warning FailedCreatePodSandBox 1m kubelet, cnv-executor-aditya-node2.example.com Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-fb7lj_default_358de615-f972-11e8-8ef1-fa163e890c9f_0(5ed7c99e6ab18cb41ca392295440790b625191fb1b2943684bdb162c0008cb73): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found Warning FailedCreatePodSandBox 1m kubelet, cnv-executor-aditya-node2.example.com Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-fb7lj_default_358de615-f972-11e8-8ef1-fa163e890c9f_0(178f7be80bc842025f565d7d14e8f3987599a2ff622110efce8a7cda2eafffa0): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found Warning FailedCreatePodSandBox 1m kubelet, cnv-executor-aditya-node2.example.com Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-fb7lj_default_358de615-f972-11e8-8ef1-fa163e890c9f_0(c60e5cbeb262a48fdec13ed95089cc85d33f833a869a957a69d10b5cd9320a82): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found Warning FailedCreatePodSandBox 58s kubelet, cnv-executor-aditya-node2.example.com Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-fb7lj_default_358de615-f972-11e8-8ef1-fa163e890c9f_0(6d23fffeaff9bb4e3492a1a71a52538f99cf3377b1526332a3c70ee82bb3989f): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found Warning FailedCreatePodSandBox 40s kubelet, cnv-executor-aditya-node2.example.com Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-fb7lj_default_358de615-f972-11e8-8ef1-fa163e890c9f_0(982ead9eda6b5edd5fe6a4b0253ae4ee0ca7939c1f246c48863e6291346459fe): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found Warning FailedCreatePodSandBox 24s kubelet, cnv-executor-aditya-node2.example.com Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-fb7lj_default_358de615-f972-11e8-8ef1-fa163e890c9f_0(6c126a5421b124b785dc3509f3c8d041f478d8b7faac025004d4d2654f7d7351): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found Warning FailedCreatePodSandBox 11s kubelet, cnv-executor-aditya-node2.example.com Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_virt-launcher-vmi-multus-ptp-fb7lj_default_358de615-f972-11e8-8ef1-fa163e890c9f_0(a4b0f2dd7f7e09ef76b2aade6d2014726c60b7de817bd88279fae419685c6b2f): Multus: Err in tearing down failed plugins: Multus: error in invoke Delegate add - "ovs": Link not found
@Aditya Please make sure you have created br1 on your nodes.
@Yan I have installed the ovs-cni manifest rpm package and deploy the yaml files and got some failures: [cloud-user@cnv-executor-aditya-master1 ~]$ sudo yum install ovs-cni-manifests ... ... Installed: ovs-cni-manifests.noarch 0:0.2.0-2 Complete! [cloud-user@cnv-executor-aditya-master1 ~]$ oc apply -f /usr/share/ovs-cni/manifests/ customresourcedefinition.apiextensions.k8s.io/network-attachment-definitions.k8s.cni.cncf.io configured clusterrole.rbac.authorization.k8s.io/multus configured clusterrolebinding.rbac.authorization.k8s.io/multus configured serviceaccount/multus unchanged configmap/multus-cni-config configured daemonset.extensions/kube-multus-ds-amd64 created daemonset.extensions/ovs-cni-plugin-amd64 created The DaemonSet "ovs-vsctl-amd64" is invalid: spec.template.metadata.labels: Invalid value: map[string]string{"app":"ovs-cni", "tier":"node"}: `selector` does not match template `labels` [cloud-user@cnv-executor-aditya-master1 ~]$ oc get pods -n kube-system NAME READY STATUS RESTARTS AGE cdi-api-556469bfbc-4xchf 1/1 Running 0 2h cdi-deployment-6c49474554-dcr9r 1/1 Running 0 2h cdi-uploadproxy-784878c6cc-tl2b9 1/1 Running 0 2h kube-multus-amd64-fwczp 1/1 Running 0 2h kube-multus-amd64-ldzck 1/1 Running 0 2h kube-multus-ds-amd64-j6hg9 0/1 ImagePullBackOff 0 23s kube-multus-ds-amd64-rbxqt 0/1 ImagePullBackOff 0 23s kube-ovs-cni-plugin-amd64-54zlv 1/1 Running 0 2h kube-ovs-cni-plugin-amd64-w7kxx 1/1 Running 0 2h master-api-cnv-executor-aditya-master1.example.com 1/1 Running 0 3h master-controllers-cnv-executor-aditya-master1.example.com 1/1 Running 0 3h master-etcd-cnv-executor-aditya-master1.example.com 1/1 Running 0 3h ovs-cni-plugin-amd64-4g54t 0/1 ImagePullBackOff 0 23s ovs-cni-plugin-amd64-fc6wn 0/1 ImagePullBackOff 0 23s ovs-vsctl-amd64-4hrz5 1/1 Running 0 2h ovs-vsctl-amd64-jvn7v 1/1 Running 0 2h virt-api-596b5678c4-tt49h 1/1 Running 0 2h virt-api-596b5678c4-zlxxt 1/1 Running 0 2h virt-controller-7f77fb7b95-zbrh5 1/1 Running 0 2h virt-controller-7f77fb7b95-zzvmt 1/1 Running 0 2h virt-handler-564rf 1/1 Running 0 2h virt-handler-bvpmt 1/1 Running 0 2h [cloud-user@cnv-executor-aditya-master1 ~]$ oc describe pod kube-multus-ds-amd64-rbxqt .... .... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Pulling 16s (x4 over 1m) kubelet, cnv-executor-aditya-node1.example.com pulling image "registry.access.redhat.com/cnv13-tech-preview/multus-cni:1.3" Warning Failed 15s (x4 over 1m) kubelet, cnv-executor-aditya-node1.example.com Failed to pull image "registry.access.redhat.com/cnv13-tech-preview/multus-cni:1.3": rpc error: code = Unknown desc = Error reading manifest 1.3 in registry.access.redhat.com/cnv13-tech-preview/multus-cni: unknown: Not Found Warning Failed 15s (x4 over 1m) kubelet, cnv-executor-aditya-node1.example.com Error: ErrImagePull Normal BackOff 4s (x6 over 1m) kubelet, cnv-executor-aditya-node1.example.com Back-off pulling image "registry.access.redhat.com/cnv13-tech-preview/multus-cni:1.3" Warning Failed 4s (x6 over 1m) kubelet, cnv-executor-aditya-node1.example.com Error: ImagePullBackOff
@Aditya I don't think the registry.access.redhat.com/cnv13-tech-preview/multus-cni:1.3 here is correct. # docker pull registry.access.redhat.com/cnv13-tech-preview/multus-cni:1.3 Trying to pull repository registry.access.redhat.com/cnv13-tech-preview/multus-cni ... unknown: Not Found
right, it should be registry.access.redhat.com/cnv-tech-preview
Please reopen if using the correct product designation does not solve.