Bug 1669474
| Summary: | [kubevirt-ansible] deployment of kubevirt-cpu-node-labeller is not idempotent | ||
|---|---|---|---|
| Product: | Container Native Virtualization (CNV) | Reporter: | Lukas Bednar <lbednar> |
| Component: | Installation | Assignee: | Karel Šimon <ksimon> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Irina Gulina <igulina> |
| Severity: | unspecified | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 1.4 | CC: | cnv-qe-bugs, ncredi, rhallise |
| Target Milestone: | --- | ||
| Target Release: | 1.4 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | kubevirt-ansible-0.12.3-1.f9f3ed4 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-03-05 14:47:27 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I have run the ansible-playbook for the second time on the same cluster as in BZ 1664276 without any issues. >> rpm -qa "*kubevirt*" kubevirt-templates-0.4.1-1.23cf119.noarch kubevirt-ansible-0.12.3-1.f9f3ed4.noarch kubevirt-virtctl-0.13.3-1.g00e92c7.b9ca79c.x86_64 |
When running kubevirt-ansible two times I fail on following error Description of problem: TASK [kubevirt-cpu-node-labeller : Create kubevirt-cpu-node-labeller] ************************************************************************************************************************* fatal: [localhost]: FAILED! => {"changed": true, "cmd": "oc create -f /tmp/kubevirt-cpu-node-labeller.yaml -n kubevirt", "delta": "0:00:00.233755", "end": "2019-01-25 06:57:38.918827", "msg": "non-zero return code", "rc": 1, "start": "2019-01-25 06:57:38.685072", "stderr": "Error from server (AlreadyExists): error when creating \"/tmp/kubevirt-cpu-node-labeller.yaml\": serviceaccounts \"kubevirt-cpu-node-labeller\" already exists\nError from server (AlreadyExists): error when creating \"/tmp/kubevirt-cpu-node-labeller.yaml\": clusterroles.rbac.authorization.k8s.io \"kubevirt-cpu-node-labeller\" already exists\nError from server (AlreadyExists): error when creating \"/tmp/kubevirt-cpu-node-labeller.yaml\": clusterrolebindings.rbac.authorization.k8s.io \"kubevirt-cpu-node-labeller\" already exists\nError from server (AlreadyExists): error when creating \"/tmp/kubevirt-cpu-node-labeller.yaml\": daemonsets.apps \"kubevirt-cpu-node-labeller\" already exists", "stderr_lines": ["Error from server (AlreadyExists): error when creating \"/tmp/kubevirt-cpu-node-labeller.yaml\": serviceaccounts \"kubevirt-cpu-node-labeller\" already exists", "Error from server (AlreadyExists): error when creating \"/tmp/kubevirt-cpu-node-labeller.yaml\": clusterroles.rbac.authorization.k8s.io \"kubevirt-cpu-node-labeller\" already exists", "Error from server (AlreadyExists): error when creating \"/tmp/kubevirt-cpu-node-labeller.yaml\": clusterrolebindings.rbac.authorization.k8s.io \"kubevirt-cpu-node-labeller\" already exists", "Error from server (AlreadyExists): error when creating \"/tmp/kubevirt-cpu-node-labeller.yaml\": daemonsets.apps \"kubevirt-cpu-node-labeller\" already exists"], "stdout": "", "stdout_lines": []} to retry, use: --limit @/usr/share/ansible/kubevirt-ansible/playbooks/kubevirt.retry PLAY RECAP ************************************************************************************************************************************************************************************ 172.16.0.17 : ok=1 changed=0 unreachable=0 failed=0 172.16.0.20 : ok=1 changed=0 unreachable=0 failed=0 172.16.0.22 : ok=1 changed=0 unreachable=0 failed=0 localhost : ok=74 changed=46 unreachable=0 failed=1 Version-Release number of selected component (if applicable): kubevirt-ansible-0.12.2-1.acde806.noarch How reproducible: 100% Steps to Reproduce: 1. run kubevirt-ansible 2. run kubevirt-ansible second time 3. Actual results: it fails on above error Expected results: it should finish successfully Additional info: [cloud-user@cnv-executor-composed-build-master1 ~]$ sudo ansible-playbook -i /home/cloud-user/kubevirt-ansible.inv -u cloud-user --private-key /home/cloud-user/.ssh/id_rsa -e@/usr/share/ansible/kubevirt-ansible/vars/all.yml -e@/usr/share/ansible/kubevirt-ansible/vars/cnv.yml -e "registry_url=brew-pulp-docker01.web.prod.ext.phx2.redhat.com:8888" -e "apb_action=provision" /usr/share/ansible/kubevirt-ansible/playbooks/kubevirt.yml PLAY [Initial configuration] ****************************************************************************************************************************************************************** TASK [Login As Super User] ******************************************************************************************************************************************************************** skipping: [localhost] TASK [Config kubernetes client binary] ******************************************************************************************************************************************************** skipping: [localhost] TASK [Config openshift client binary] ********************************************************************************************************************************************************* ok: [localhost] PLAY [Initial configuration] ****************************************************************************************************************************************************************** TASK [Login As Super User] ******************************************************************************************************************************************************************** skipping: [localhost] TASK [Config kubernetes client binary] ******************************************************************************************************************************************************** skipping: [localhost] TASK [Config openshift client binary] ********************************************************************************************************************************************************* ok: [localhost] PLAY [nodes masters] ************************************************************************************************************************************************************************** TASK [Gathering Facts] ************************************************************************************************************************************************************************ ok: [172.16.0.22] ok: [172.16.0.17] ok: [172.16.0.20] TASK [remove multus config from nodes on deprovisioning] ************************************************************************************************************************************** skipping: [172.16.0.20] => (item=/etc/cni/net.d/00-multus.conf) skipping: [172.16.0.20] => (item=/etc/cni/net.d/multus.d) skipping: [172.16.0.22] => (item=/etc/cni/net.d/00-multus.conf) skipping: [172.16.0.22] => (item=/etc/cni/net.d/multus.d) skipping: [172.16.0.17] => (item=/etc/cni/net.d/00-multus.conf) skipping: [172.16.0.17] => (item=/etc/cni/net.d/multus.d) TASK [make sure ovs is installed] ************************************************************************************************************************************************************* skipping: [172.16.0.20] skipping: [172.16.0.22] skipping: [172.16.0.17] TASK [enable and start OVS] ******************************************************************************************************************************************************************* skipping: [172.16.0.20] skipping: [172.16.0.22] skipping: [172.16.0.17] TASK [Fix SELinux labels for /var/lib/kubelet/device-plugins/] ******************************************************************************************************************************** skipping: [172.16.0.20] skipping: [172.16.0.22] skipping: [172.16.0.17] PLAY [Deploy network roles] ******************************************************************************************************************************************************************* TASK [network-multus : include_tasks] ********************************************************************************************************************************************************* included: /usr/share/ansible/kubevirt-ansible/roles/network-multus/tasks/provision.yml for localhost TASK [network-multus : Check if namespace "kube-system" exists] ******************************************************************************************************************************* changed: [localhost] TASK [network-multus : Create kube-system namespace] ****************************************************************************************************************************************** skipping: [localhost] TASK [network-multus : openshift cni config] ************************************************************************************************************************************************** ok: [localhost] TASK [network-multus : kubernetes cni config] ************************************************************************************************************************************************* skipping: [localhost] TASK [network-multus : Render multus deployment yaml] ***************************************************************************************************************************************** ok: [localhost] TASK [network-multus : Create multus Resources] *********************************************************************************************************************************************** changed: [localhost] TASK [network-multus : Render cni plugins deployment yaml] ************************************************************************************************************************************ skipping: [localhost] TASK [network-multus : Create cni plugins Resources] ****************************************************************************************************************************************** skipping: [localhost] TASK [network-multus : Render OVS deployment yaml] ******************************************************************************************************************************************** ok: [localhost] TASK [network-multus : Create ovs Resources] ************************************************************************************************************************************************** changed: [localhost] TASK [network-multus : Render ovs-vsctl deployment yaml] ************************************************************************************************************************************** ok: [localhost] TASK [network-multus : Create ovs-vsctl resources] ******************************************************************************************************************************************** changed: [localhost] TASK [network-multus : Render SR-IOV DP deployment yaml] ************************************************************************************************************************************** skipping: [localhost] TASK [network-multus : Create SR-IOV DP resources] ******************************************************************************************************************************************** skipping: [localhost] TASK [network-multus : Render SR-IOV CNI deployment yaml] ************************************************************************************************************************************* skipping: [localhost] TASK [network-multus : Create SR-IOV CNI resources] ******************************************************************************************************************************************* skipping: [localhost] TASK [network-multus : Render SR-IOV network CRD yaml] **************************************************************************************************************************************** skipping: [localhost] TASK [network-multus : Create SR-IOV network CRD] ********************************************************************************************************************************************* skipping: [localhost] TASK [network-multus : Wait until multus is running] ****************************************************************************************************************************************** FAILED - RETRYING: Wait until multus is running (24 retries left). changed: [localhost] TASK [network-multus : Wait until CNI plugins are running] ************************************************************************************************************************************ skipping: [localhost] TASK [network-multus : Wait until OVS Plugin is running] ************************************************************************************************************************************** changed: [localhost] TASK [network-multus : Wait until SR-IOV DP plugin is running] ******************************************************************************************************************************** skipping: [localhost] TASK [network-multus : Wait until SR-IOV CNI plugin is running] ******************************************************************************************************************************* skipping: [localhost] TASK [skydive : include_tasks] **************************************************************************************************************************************************************** skipping: [localhost] PLAY [Deploy kubevirt role] ******************************************************************************************************************************************************************* TASK [kubevirt : include_tasks] *************************************************************************************************************************************************************** included: /usr/share/ansible/kubevirt-ansible/roles/kubevirt/tasks/provision.yml for localhost TASK [kubevirt : Check if kubevirt exists] **************************************************************************************************************************************************** changed: [localhost] TASK [kubevirt : Create kubevirt namespace] *************************************************************************************************************************************************** skipping: [localhost] TASK [kubevirt : Render KubeVirt operator] **************************************************************************************************************************************************** ok: [localhost] TASK [kubevirt : Create KubeVirt Operator] **************************************************************************************************************************************************** changed: [localhost] TASK [kubevirt : Enable kubevirt feature gates] *********************************************************************************************************************************************** changed: [localhost] TASK [kubevirt : Render KubeVirt resources] *************************************************************************************************************************************************** ok: [localhost] TASK [kubevirt : Create KubeVirt Resources] *************************************************************************************************************************************************** changed: [localhost] TASK [kubevirt : Give the kubevirt-operator user privilages] ********************************************************************************************************************************** changed: [localhost] TASK [kubevirt : Wait until virt-operator is running] ***************************************************************************************************************************************** FAILED - RETRYING: Wait until virt-operator is running (24 retries left). FAILED - RETRYING: Wait until virt-operator is running (23 retries left). changed: [localhost] TASK [kubevirt : Wait until virt-api is running] ********************************************************************************************************************************************** FAILED - RETRYING: Wait until virt-api is running (24 retries left). FAILED - RETRYING: Wait until virt-api is running (23 retries left). changed: [localhost] TASK [kubevirt : Wait until virt-controller is running] *************************************************************************************************************************************** changed: [localhost] TASK [kubevirt : Wait until virt-handler is running] ****************************************************************************************************************************************** changed: [localhost] TASK [kubevirt : Debug vmis] ****************************************************************************************************************************************************************** changed: [localhost] TASK [kubevirt : debug] *********************************************************************************************************************************************************************** ok: [localhost] => { "crds.stdout": "NAME CREATED AT\nalertmanagers.monitoring.coreos.com 2019-01-25T08:30:45Z\nbundlebindings.automationbroker.io 2019-01-25T08:47:51Z\nbundleinstances.automationbroker.io 2019-01-25T08:47:52Z\nbundles.automationbroker.io 2019-01-25T08:47:53Z\ncdis.cdi.kubevirt.io 2019-01-25T09:00:25Z\ndatavolumes.cdi.kubevirt.io 2019-01-25T09:00:38Z\nkubevirts.kubevirt.io 2019-01-25T11:56:36Z\nkwebuis.kubevirt.io 2019-01-25T08:59:39Z\nnetwork-attachment-definitions.k8s.cni.cncf.io 2019-01-25T11:56:21Z\nprometheuses.monitoring.coreos.com 2019-01-25T08:30:45Z\nprometheusrules.monitoring.coreos.com 2019-01-25T08:30:45Z\nservicemonitors.monitoring.coreos.com 2019-01-25T08:30:45Z\nvirtualmachineinstancemigrations.kubevirt.io 2019-01-25T11:56:46Z\nvirtualmachineinstancepresets.kubevirt.io 2019-01-25T11:56:46Z\nvirtualmachineinstancereplicasets.kubevirt.io 2019-01-25T11:56:46Z\nvirtualmachineinstances.kubevirt.io 2019-01-25T11:56:46Z\nvirtualmachines.kubevirt.io 2019-01-25T11:56:46Z" } PLAY [Initial configuration] ****************************************************************************************************************************************************************** TASK [Login As Super User] ******************************************************************************************************************************************************************** skipping: [localhost] TASK [Config kubernetes client binary] ******************************************************************************************************************************************************** skipping: [localhost] TASK [Config openshift client binary] ********************************************************************************************************************************************************* ok: [localhost] PLAY [Kubevirt Web UI] ************************************************************************************************************************************************************************ TASK [kubevirt_web_ui : include_tasks] ******************************************************************************************************************************************************** skipping: [localhost] TASK [kubevirt_web_ui : include_tasks] ******************************************************************************************************************************************************** skipping: [localhost] TASK [kubevirt_web_ui : include_tasks] ******************************************************************************************************************************************************** included: /usr/share/ansible/kubevirt-ansible/roles/kubevirt_web_ui/tasks/provision.operator.yml for localhost TASK [kubevirt_web_ui : set_fact] ************************************************************************************************************************************************************* ok: [localhost] TASK [kubevirt_web_ui : Make temp directory] ************************************************************************************************************************************************** ok: [localhost] TASK [kubevirt_web_ui : Check if project for kubevirt-web-ui exists] ************************************************************************************************************************** changed: [localhost] TASK [kubevirt_web_ui : Create project for kubevirt-web-ui] *********************************************************************************************************************************** skipping: [localhost] TASK [kubevirt_web_ui : Change project for kubevirt-web-ui] *********************************************************************************************************************************** changed: [localhost] TASK [kubevirt_web_ui : Check if openshift-console project exists] **************************************************************************************************************************** changed: [localhost] TASK [kubevirt_web_ui : Create service account] *********************************************************************************************************************************************** changed: [localhost] TASK [kubevirt_web_ui : Add SCC to the service account] *************************************************************************************************************************************** changed: [localhost] TASK [kubevirt_web_ui : Add roles] ************************************************************************************************************************************************************ changed: [localhost] TASK [kubevirt_web_ui : Render roles bindings for openshift-console] ************************************************************************************************************************** changed: [localhost] TASK [kubevirt_web_ui : Apply roles for openshift-console] ************************************************************************************************************************************ changed: [localhost] TASK [kubevirt_web_ui : Render role bindings] ************************************************************************************************************************************************* changed: [localhost] TASK [kubevirt_web_ui : Apply role bindings] ************************************************************************************************************************************************** changed: [localhost] TASK [kubevirt_web_ui : Render role bindings for openshift-console] *************************************************************************************************************************** changed: [localhost] TASK [kubevirt_web_ui : Apply role bindings for openshift-console] **************************************************************************************************************************** changed: [localhost] TASK [kubevirt_web_ui : Add Custom Resource Definition for operator] ************************************************************************************************************************** changed: [localhost] TASK [kubevirt_web_ui : Render Operator yaml] ************************************************************************************************************************************************* changed: [localhost] TASK [kubevirt_web_ui : Deploy Web UI Operator] *********************************************************************************************************************************************** changed: [localhost] TASK [kubevirt_web_ui : include_tasks] ******************************************************************************************************************************************************** included: /usr/share/ansible/kubevirt-ansible/roles/kubevirt_web_ui/tasks/provision.web-ui.yml for localhost TASK [kubevirt_web_ui : Make temp directory] ************************************************************************************************************************************************** ok: [localhost] TASK [kubevirt_web_ui : Change project for kubevirt-web-ui] *********************************************************************************************************************************** changed: [localhost] TASK [kubevirt_web_ui : Copy custom resource yaml to temp directory] ************************************************************************************************************************** changed: [localhost] TASK [kubevirt_web_ui : Set kubevirt-web-ui version] ****************************************************************************************************************************************** changed: [localhost] TASK [kubevirt_web_ui : Set kubevirt-web-ui registry_url] ************************************************************************************************************************************* changed: [localhost] TASK [kubevirt_web_ui : Set kubevirt-web-ui registry_namespace] ******************************************************************************************************************************* changed: [localhost] TASK [kubevirt_web_ui : Set kubevirt-web-ui branding] ***************************************************************************************************************************************** changed: [localhost] TASK [kubevirt_web_ui : Set public_master_hostname] ******************************************************************************************************************************************* skipping: [localhost] TASK [kubevirt_web_ui : Set openshift_master_default_subdomain] ******************************************************************************************************************************* skipping: [localhost] TASK [kubevirt_web_ui : Create operator custom resource to (un)deploy the Web UI] ************************************************************************************************************* changed: [localhost] TASK [kubevirt_web_ui : Wait until Web UI is ready] ******************************************************************************************************************************************* changed: [localhost] TASK [kubevirt_web_ui : Wait until Web UI is undeployed] ************************************************************************************************************************************** skipping: [localhost] PLAY [Initial configuration] ****************************************************************************************************************************************************************** TASK [Login As Super User] ******************************************************************************************************************************************************************** skipping: [localhost] TASK [Config kubernetes client binary] ******************************************************************************************************************************************************** skipping: [localhost] TASK [Config openshift client binary] ********************************************************************************************************************************************************* ok: [localhost] PLAY [Deploy cdi role] ************************************************************************************************************************************************************************ TASK [cdi : include_tasks] ******************************************************************************************************************************************************************** included: /usr/share/ansible/kubevirt-ansible/roles/cdi/tasks/provision.yml for localhost TASK [cdi : Check if namespace cdi exists] **************************************************************************************************************************************************** changed: [localhost] TASK [cdi : Create cdi namespace using kubectl] *********************************************************************************************************************************************** skipping: [localhost] TASK [cdi : Create cdi namespace using oc] **************************************************************************************************************************************************** skipping: [localhost] TASK [cdi : Render CDI operator] ************************************************************************************************************************************************************** ok: [localhost] TASK [cdi : Create CDI Operator] ************************************************************************************************************************************************************** changed: [localhost] TASK [cdi : Wait until cdi-operator is running] *********************************************************************************************************************************************** changed: [localhost] TASK [cdi : Render CDI Upload Proxy Route] **************************************************************************************************************************************************** ok: [localhost] TASK [cdi : Create CDI Upload Proxy Route] **************************************************************************************************************************************************** changed: [localhost] TASK [cdi : Render CDI operator resources] **************************************************************************************************************************************************** ok: [localhost] TASK [cdi : Create CDI operator resources] **************************************************************************************************************************************************** changed: [localhost] TASK [cdi : Wait for CDI operator to report that everything is running] *********************************************************************************************************************** changed: [localhost] PLAY [Initial configuration] ****************************************************************************************************************************************************************** TASK [Login As Super User] ******************************************************************************************************************************************************************** skipping: [localhost] TASK [Config kubernetes client binary] ******************************************************************************************************************************************************** skipping: [localhost] TASK [Config openshift client binary] ********************************************************************************************************************************************************* ok: [localhost] PLAY [Deploy kubevirt-ssp role] *************************************************************************************************************************************************************** TASK [kubevirt-ssp : include_tasks] *********************************************************************************************************************************************************** included: /usr/share/ansible/kubevirt-ansible/roles/kubevirt-ssp/tasks/provision.yml for localhost TASK [kubevirt-ssp : Check that common-templates.yaml still exists in /tmp] ******************************************************************************************************************* ok: [localhost] TASK [kubevirt-ssp : Copy common templates yaml to temp directory] **************************************************************************************************************************** skipping: [localhost] TASK [kubevirt-ssp : Create Kubevirt Template] ************************************************************************************************************************************************ changed: [localhost] PLAY [Initial configuration] ****************************************************************************************************************************************************************** TASK [Login As Super User] ******************************************************************************************************************************************************************** skipping: [localhost] TASK [Config kubernetes client binary] ******************************************************************************************************************************************************** skipping: [localhost] TASK [Config openshift client binary] ********************************************************************************************************************************************************* ok: [localhost] PLAY [Deploy kubevirt-cpu-node-labeller role] ************************************************************************************************************************************************* TASK [kubevirt-cpu-node-labeller : include_tasks] ********************************************************************************************************************************************* included: /usr/share/ansible/kubevirt-ansible/roles/kubevirt-cpu-node-labeller/tasks/provision.yml for localhost TASK [kubevirt-cpu-node-labeller : Copy kubevirt-cpu-node-labeller.yaml to temp directory] **************************************************************************************************** ok: [localhost] TASK [kubevirt-cpu-node-labeller : Create kubevirt-cpu-node-labeller] ************************************************************************************************************************* fatal: [localhost]: FAILED! => {"changed": true, "cmd": "oc create -f /tmp/kubevirt-cpu-node-labeller.yaml -n kubevirt", "delta": "0:00:00.233755", "end": "2019-01-25 06:57:38.918827", "msg": "non-zero return code", "rc": 1, "start": "2019-01-25 06:57:38.685072", "stderr": "Error from server (AlreadyExists): error when creating \"/tmp/kubevirt-cpu-node-labeller.yaml\": serviceaccounts \"kubevirt-cpu-node-labeller\" already exists\nError from server (AlreadyExists): error when creating \"/tmp/kubevirt-cpu-node-labeller.yaml\": clusterroles.rbac.authorization.k8s.io \"kubevirt-cpu-node-labeller\" already exists\nError from server (AlreadyExists): error when creating \"/tmp/kubevirt-cpu-node-labeller.yaml\": clusterrolebindings.rbac.authorization.k8s.io \"kubevirt-cpu-node-labeller\" already exists\nError from server (AlreadyExists): error when creating \"/tmp/kubevirt-cpu-node-labeller.yaml\": daemonsets.apps \"kubevirt-cpu-node-labeller\" already exists", "stderr_lines": ["Error from server (AlreadyExists): error when creating \"/tmp/kubevirt-cpu-node-labeller.yaml\": serviceaccounts \"kubevirt-cpu-node-labeller\" already exists", "Error from server (AlreadyExists): error when creating \"/tmp/kubevirt-cpu-node-labeller.yaml\": clusterroles.rbac.authorization.k8s.io \"kubevirt-cpu-node-labeller\" already exists", "Error from server (AlreadyExists): error when creating \"/tmp/kubevirt-cpu-node-labeller.yaml\": clusterrolebindings.rbac.authorization.k8s.io \"kubevirt-cpu-node-labeller\" already exists", "Error from server (AlreadyExists): error when creating \"/tmp/kubevirt-cpu-node-labeller.yaml\": daemonsets.apps \"kubevirt-cpu-node-labeller\" already exists"], "stdout": "", "stdout_lines": []} to retry, use: --limit @/usr/share/ansible/kubevirt-ansible/playbooks/kubevirt.retry PLAY RECAP ************************************************************************************************************************************************************************************ 172.16.0.17 : ok=1 changed=0 unreachable=0 failed=0 172.16.0.20 : ok=1 changed=0 unreachable=0 failed=0 172.16.0.22 : ok=1 changed=0 unreachable=0 failed=0 localhost : ok=74 changed=46 unreachable=0 failed=1