Description of problem: When trying to deploy kubevirt using kubevirt-apb, apb container never start. When taking closer look I see following error: Warning FailedCreatePodSandBox 0s (x7 over 1m) kubelet, cnv-executor-lbednar-node1.example.com (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2_brew-virtualization-prov-jf52z_313031b9-d918-11e8-8a5a-fa163e4f7c37_0(47e20877e0b21c1cd89cef5556c5b0050c4004a3099301d8b9e025eb50b486a8): Multus: Err in loading K8s Delegates k8s args: Multus: Err in getting k8s network from pod: getPodNetworkAnnotation: failed to query the pod bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2 in out of cluster comm: Unauthorized [root@cnv-executor-lbednar-master1 ~]# oc logs -f -n brew-virtualization-prov-jf52z bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2 Error from server (BadRequest): container "apb" in pod "bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2" is waiting to start: ContainerCreating Version-Release number of selected component (if applicable): kubevirt-apb-v1.3.0-4 How reproducible: 100% Steps to Reproduce: 1. Create kubevirt-apb serviceinstance 2. Watch how kubevirt-apb container is stucked in ContainerCreating state 3. Actual results: can not deploy kubevirt using kubevirt-apb Expected results: kubevirt gets deployed via kubevirt-apb Additional info: [root@cnv-executor-lbednar-master1 ~]# cat kubevirt-apb.yml --- apiVersion: servicecatalog.k8s.io/v1beta1 kind: ServiceInstance metadata: name: kubevirt namespace: kube-system spec: clusterServiceClassExternalName: brew-virtualization clusterServicePlanExternalName: default parameters: admin_user: "test_admin" admin_password: "123456" docker_tag: "1.3" registry_url: "brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888" registry_namespace: "cnv13-tech-preview" [root@cnv-executor-lbednar-master1 ~]# oc get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE brew-virtualization-prov-jf52z bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2 0/1 ContainerCreating 0 1m [root@cnv-executor-lbednar-master1 ~]# oc logs -f -n brew-virtualization-prov-jf52z bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2 Error from server (BadRequest): container "apb" in pod "bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2" is waiting to start: ContainerCreating [root@cnv-executor-lbednar-master1 ~]# oc describe pods -n brew-virtualization-prov-jf52z bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2 Name: bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2 Namespace: brew-virtualization-prov-jf52z Priority: 0 PriorityClassName: <none> Node: cnv-executor-lbednar-node1.example.com/172.16.0.16 Start Time: Fri, 26 Oct 2018 08:10:55 -0400 Labels: bundle-action=provision bundle-fqname=brew-virtualization bundle-pod-name=bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2 Annotations: openshift.io/scc=restricted Status: Pending IP: Containers: apb: Container ID: Image: brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cnv13-tech-preview/kubevirt-apb:latest Image ID: Port: <none> Host Port: <none> Args: provision --extra-vars {"_apb_last_requesting_user":"test_admin","_apb_plan_id":"default","_apb_service_class_id":"478d37510d55e47a89cfbf1d10f92b7d","_apb_service_instance_id":"3098ad93-d918-11e8-9675-0a580a800012","admin_password":"123456","admin_user":"test_admin","cluster":"openshift","docker_tag":"1.3","namespace":"kube-system","registry_namespace":"cnv13-tech-preview","registry_url":"brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888"} State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment: POD_NAME: bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2 (v1:metadata.name) POD_NAMESPACE: brew-virtualization-prov-jf52z (v1:metadata.namespace) Mounts: /var/run/secrets/kubernetes.io/serviceaccount from bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2-token-2hf6s (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2-token-2hf6s: Type: Secret (a volume populated by a Secret) SecretName: bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2-token-2hf6s Optional: false QoS Class: BestEffort Node-Selectors: node-role.kubernetes.io/compute=true Tolerations: <none> Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedCreatePodSandBox 3m kubelet, cnv-executor-lbednar-node1.example.com Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2_brew-virtualization-prov-jf52z_313031b9-d918-11e8-8a5a-fa163e4f7c37_0(af67f17ae927602b7ff80fb59a4c8d1db8603f16c0d7f381c78b9f74f8de06f6): Multus: Err in loading K8s Delegates k8s args: Multus: Err in getting k8s network from pod: getPodNetworkAnnotation: failed to query the pod bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2 in out of cluster comm: Unauthorized Normal Scheduled 3m default-scheduler Successfully assigned brew-virtualization-prov-jf52z/bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2 to cnv-executor-lbednar-node1.example.com Warning FailedCreatePodSandBox 3m kubelet, cnv-executor-lbednar-node1.example.com Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2_brew-virtualization-prov-jf52z_313031b9-d918-11e8-8a5a-fa163e4f7c37_0(c3fca24bb21cf72417d37b48a437686098466acc467cd8eef8b58ffa1fa5827d): Multus: Err in loading K8s Delegates k8s args: Multus: Err in getting k8s network from pod: getPodNetworkAnnotation: failed to query the pod bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2 in out of cluster comm: Unauthorized Warning FailedCreatePodSandBox 2m kubelet, cnv-executor-lbednar-node1.example.com Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2_brew-virtualization-prov-jf52z_313031b9-d918-11e8-8a5a-fa163e4f7c37_0(0d18a49c65c79b9351a2ef5d91b42e62161c2ef3c19f28a82a57662aa30703f7): Multus: Err in loading K8s Delegates k8s args: Multus: Err in getting k8s network from pod: getPodNetworkAnnotation: failed to query the pod bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2 in out of cluster comm: Unauthorized Warning FailedCreatePodSandBox 2m kubelet, cnv-executor-lbednar-node1.example.com Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2_brew-virtualization-prov-jf52z_313031b9-d918-11e8-8a5a-fa163e4f7c37_0(f6162b4fd09987422e671cde93bd2e9bd219ccc108d16d40c49cd7ab6d752fbb): Multus: Err in loading K8s Delegates k8s args: Multus: Err in getting k8s network from pod: getPodNetworkAnnotation: failed to query the pod bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2 in out of cluster comm: Unauthorized Warning FailedCreatePodSandBox 2m kubelet, cnv-executor-lbednar-node1.example.com Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2_brew-virtualization-prov-jf52z_313031b9-d918-11e8-8a5a-fa163e4f7c37_0(e0125210ccc3a1be3785228d802d981bcd3cf51c92131521c19611c95faf8388): Multus: Err in loading K8s Delegates k8s args: Multus: Err in getting k8s network from pod: getPodNetworkAnnotation: failed to query the pod bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2 in out of cluster comm: Unauthorized Warning FailedCreatePodSandBox 2m kubelet, cnv-executor-lbednar-node1.example.com Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2_brew-virtualization-prov-jf52z_313031b9-d918-11e8-8a5a-fa163e4f7c37_0(ab4779c05367c4866675682a8bda45b8a27485340169e9b4601b4492f31a1368): Multus: Err in loading K8s Delegates k8s args: Multus: Err in getting k8s network from pod: getPodNetworkAnnotation: failed to query the pod bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2 in out of cluster comm: Unauthorized Warning FailedCreatePodSandBox 2m kubelet, cnv-executor-lbednar-node1.example.com Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2_brew-virtualization-prov-jf52z_313031b9-d918-11e8-8a5a-fa163e4f7c37_0(f45b7c0f2d7008ea4a5f6b3e60a11ce1268dfab93498f05dce4883aeff7c45dc): Multus: Err in loading K8s Delegates k8s args: Multus: Err in getting k8s network from pod: getPodNetworkAnnotation: failed to query the pod bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2 in out of cluster comm: Unauthorized Warning FailedCreatePodSandBox 1m kubelet, cnv-executor-lbednar-node1.example.com Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2_brew-virtualization-prov-jf52z_313031b9-d918-11e8-8a5a-fa163e4f7c37_0(ac877d20db4f8c9a520952995b178d29606d53e7e453061b5216dc69d9c29ef2): Multus: Err in loading K8s Delegates k8s args: Multus: Err in getting k8s network from pod: getPodNetworkAnnotation: failed to query the pod bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2 in out of cluster comm: Unauthorized Warning FailedCreatePodSandBox 1m kubelet, cnv-executor-lbednar-node1.example.com Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2_brew-virtualization-prov-jf52z_313031b9-d918-11e8-8a5a-fa163e4f7c37_0(5b6041887c83f4959c88b62b1feea2e824de299e19e599798c9f2b3779aa0a6f): Multus: Err in loading K8s Delegates k8s args: Multus: Err in getting k8s network from pod: getPodNetworkAnnotation: failed to query the pod bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2 in out of cluster comm: Unauthorized Warning FailedCreatePodSandBox 0s (x7 over 1m) kubelet, cnv-executor-lbednar-node1.example.com (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2_brew-virtualization-prov-jf52z_313031b9-d918-11e8-8a5a-fa163e4f7c37_0(47e20877e0b21c1cd89cef5556c5b0050c4004a3099301d8b9e025eb50b486a8): Multus: Err in loading K8s Delegates k8s args: Multus: Err in getting k8s network from pod: getPodNetworkAnnotation: failed to query the pod bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2 in out of cluster comm: Unauthorized [root@cnv-executor-lbednar-master1 ~]# oc logs -f -n brew-virtualization-prov-jf52z bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2 Error from server (BadRequest): container "apb" in pod "bundle-da4d395b-2a8c-445a-8fdb-15e0c1b14fe2" is waiting to start: ContainerCreating
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=798617
brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888/cnv-tech-preview/kubevirt-apb:v3.11-1
Trying new image and, it failed for me. Unfortunately I didn't catch provision error, but at least I caught deprovision error: [root@cnv-executor-lbednar-master1 ~]# oc logs -n brew2-virtualization-depr-fnhtr bundle-f8815c60-5663-4789-a4b9-2b777a989915 DEPRECATED: APB playbooks should be stored at /opt/apb/project PLAY [Deprovision KubeVirt] **************************************************** TASK [ansible.kubernetes-modules : Install latest openshift client] ************ skipping: [localhost] TASK [ansibleplaybookbundle.asb-modules : debug] ******************************* skipping: [localhost] PLAY [all] ********************************************************************* TASK [Identify cluster] ******************************************************** changed: [localhost] TASK [Set cluster variable] **************************************************** [WARNING]: when statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Found: {{ result.rc }} == 0 ok: [localhost] TASK [Login As Super User] ***************************************************** changed: [localhost] PLAY [masters[0]] ************************************************************** TASK [network-multus : include_tasks] ****************************************** included: /etc/ansible/roles/kubevirt-ansible/roles/network-multus/tasks/deprovision.yml for localhost TASK [network-multus : openshift cni config] *********************************** ok: [localhost] TASK [network-multus : kubernetes cni config] ********************************** skipping: [localhost] TASK [network-multus : Render multus deployment yaml] ************************** changed: [localhost] TASK [network-multus : Delete multus Resources] ******************************** changed: [localhost] TASK [network-multus : Render cni plugins deployment yaml] ********************* changed: [localhost] TASK [network-multus : Delete cni plugins Resources] *************************** changed: [localhost] TASK [network-multus : Render OVS plugin deployment yaml] ********************** changed: [localhost] TASK [network-multus : Delete OVS plugin Resources] **************************** changed: [localhost] TASK [network-multus : Render ovs-vsctl deployment yaml] *********************** changed: [localhost] TASK [network-multus : Delete ovs-vsctl Resources] ***************************** changed: [localhost] TASK [skydive : include_tasks] ************************************************* skipping: [localhost] PLAY [masters[0]] ************************************************************** TASK [kubevirt : include_tasks] ************************************************ included: /etc/ansible/roles/kubevirt-ansible/roles/kubevirt/tasks/deprovision.yml for localhost TASK [kubevirt : Check that demo-content.yaml still exists in /tmp] ************ ok: [localhost] TASK [kubevirt : Check for demo-content.yaml template in /etc/ansible/roles/kubevirt-ansible/roles/kubevirt/templates] *** ok: [localhost] TASK [kubevirt : Download Demo Content] **************************************** fatal: [localhost]: FAILED! => {"changed": false, "msg": "Destination /etc/ansible/roles/kubevirt-ansible/roles/kubevirt/templates is not writable"} PLAY RECAP ********************************************************************* localhost : ok=16 changed=10 unreachable=0 failed=1
kubevirt-apb-v3.11-4 is working as expected, just web-ui doesn't seems to be deployed, but it is topic for other bug.
New build kubevirt-apb-v3.11-5 brought this issue back in ... Warning FailedCreatePodSandBox 3m (x233 over 56m) kubelet, cnv-executor-lbednar-node2.example.com (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_bundle-55c217c0-5f34-409b-8df2-3a757ddf2034_brew2-virtualization-depr-95s5f_5c4cf3e9-e806-11e8-9ca0-fa163e815e5a_0(1a259e60bcc3315c007ecb1cf7c8c47276e0f27db394563fadd5c247f5b571d7): Multus: Err in loading K8s Delegates k8s args: Multus: Err in getting k8s network from pod: getPodNetworkAnnotation: failed to query the pod bundle-55c217c0-5f34-409b-8df2-3a757ddf2034 in out of cluster comm: Unauthorized
There was a bug opened about for OCP Networking https://bugzilla.redhat.com/show_bug.cgi?id=1651693 .