Bug 1793603
Summary: | kubevirt-ssp-operator's ansible-playbook fails in template validator task | ||||||
---|---|---|---|---|---|---|---|
Product: | Container Native Virtualization (CNV) | Reporter: | Oren Cohen <ocohen> | ||||
Component: | SSP | Assignee: | sgott | ||||
Status: | CLOSED ERRATA | QA Contact: | Ruth Netser <rnetser> | ||||
Severity: | medium | Docs Contact: | |||||
Priority: | unspecified | ||||||
Version: | 2.1.0 | CC: | cnv-qe-bugs, fdeutsch, ksimon | ||||
Target Milestone: | --- | ||||||
Target Release: | 2.3.0 | ||||||
Hardware: | All | ||||||
OS: | Linux | ||||||
Whiteboard: | |||||||
Fixed In Version: | kubevirt-ssp-operator-container-v2.3.0-8 | Doc Type: | If docs needed, set a value | ||||
Doc Text: | Story Points: | --- | |||||
Clone Of: | Environment: | ||||||
Last Closed: | 2020-05-04 19:10:37 UTC | Type: | Bug | ||||
Regression: | --- | Mount Type: | --- | ||||
Documentation: | --- | CRM: | |||||
Verified Versions: | Category: | --- | |||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
Cloudforms Team: | --- | Target Upstream Version: | |||||
Embargoed: | |||||||
Attachments: |
|
This PR will fix it: https://github.com/MarSik/kubevirt-ssp-operator/pull/142 Verified: OCP 4.4.0-rc.2 CNV 2.3 (v0.26.3) 1. SSP operator deploys template-validator ------------------------------------------ virt-template-validator-5bfb67ff94-6chct 1/1 Running 0 93m virt-template-validator-5bfb67ff94-qt54r 1/1 Running 0 86m 2. No errors in SSP oeprator log -------------------------------- {"level":"info","ts":1584625326.419964,"logger":"logging_event_handler","msg":"[playbook task]","name":"template-validator-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"kubevirt.io/v1, Kind=KubevirtTemplateValidator","event_type":"playbook_on_task_start","job":"2740103009342231109","EventData.Name":"Gathering Facts"} {"level":"info","ts":1584625331.1642094,"logger":"logging_event_handler","msg":"[playbook debug]","name":"template-validator-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"kubevirt.io/v1, Kind=KubevirtTemplateValidator","event_type":"runner_on_ok","job":"2740103009342231109","EventData.TaskArgs":""} {"level":"info","ts":1584625331.183627,"logger":"logging_event_handler","msg":"[playbook task]","name":"template-validator-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"kubevirt.io/v1, Kind=KubevirtTemplateValidator","event_type":"playbook_on_task_start","job":"2740103009342231109","EventData.Name":"KubevirtTemplateValidator : Set template:view role"} {"level":"info","ts":1584625332.967145,"logger":"logging_event_handler","msg":"[playbook task]","name":"template-validator-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"kubevirt.io/v1, Kind=KubevirtTemplateValidator","event_type":"playbook_on_task_start","job":"2740103009342231109","EventData.Name":"KubevirtTemplateValidator : Create the service"} {"level":"info","ts":1584625338.4236267,"logger":"logging_event_handler","msg":"[playbook task]","name":"template-validator-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"kubevirt.io/v1, Kind=KubevirtTemplateValidator","event_type":"playbook_on_task_start","job":"2740103009342231109","EventData.Name":"KubevirtTemplateValidator : Register the webhook"} {"level":"info","ts":1584625340.5451968,"logger":"logging_event_handler","msg":"[playbook task]","name":"template-validator-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"kubevirt.io/v1, Kind=KubevirtTemplateValidator","event_type":"playbook_on_task_start","job":"2740103009342231109","EventData.Name":"KubevirtTemplateValidator : Set progressing condition"} {"level":"info","ts":1584625342.7724156,"logger":"logging_event_handler","msg":"[playbook task]","name":"template-validator-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"kubevirt.io/v1, Kind=KubevirtTemplateValidator","event_type":"playbook_on_task_start","job":"2740103009342231109","EventData.Name":"KubevirtTemplateValidator : Wait for the template-validator to start"} {"level":"info","ts":1584625344.5061178,"logger":"logging_event_handler","msg":"[playbook task]","name":"template-validator-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"kubevirt.io/v1, Kind=KubevirtTemplateValidator","event_type":"playbook_on_task_start","job":"2740103009342231109","EventData.Name":"KubevirtTemplateValidator : Set available condition"} {"level":"info","ts":1584625345.9363396,"logger":"logging_event_handler","msg":"[playbook task]","name":"template-validator-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"kubevirt.io/v1, Kind=KubevirtTemplateValidator","event_type":"playbook_on_task_start","job":"2740103009342231109","EventData.Name":"KubevirtTemplateValidator : Set degraded condition"} {"level":"info","ts":1584625347.7245126,"logger":"runner","msg":"Ansible-runner exited successfully","job":"2740103009342231109","name":"template-validator-kubevirt-hyperconverged","namespace":"openshift-cnv"} 3. Validate apiVersion in template-validator deplyment ------------------------------------------------------ $ oc get deployment -n openshift-cnv virt-template-validator -oyaml|grep -A2 app apiVersion: apps/v1 kind: Deployment 4. Validate template-validator pods' cpu: ----------------------------------------- $ for i in `oc get pod -n openshift-cnv -oname|grep valida|cut -d'/' -f2`; do echo $i; kubectl top pod $i --containers -n openshift-cnv; done virt-template-validator-5bfb67ff94-6chct POD NAME CPU(cores) MEMORY(bytes) virt-template-validator-5bfb67ff94-6chct webhook 1m 20Mi virt-template-validator-5bfb67ff94-qt54r POD NAME CPU(cores) MEMORY(bytes) virt-template-validator-5bfb67ff94-qt54r webhook 1m 16Mi Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:2011 |
Created attachment 1654301 [details] ssp-operator logs, from the beginning. Description of problem: ssp-operator is running an ansible-playbook that continuously crashing at the template-validator task, thus consuming high amount of cpu usage ~ 1.1 CPU core on average. Version-Release number of selected component (if applicable): SSP version: registry.redhat.io/container-native-virtualization/kubevirt-ssp-operator:v2.1.0-22 Openshift version: 4.4.0-0.ci-2020-01-19-121550 HCO version: registry.redhat.io/container-native-virtualization/hyperconverged-cluster-operator:v2.1.0-22 How reproducible: 100% Steps to Reproduce: 1. Deploy CNV 2.1 on Openshift 4.4 2. 3. Actual results: ssp-operator consumes three order of magnitude more CPU than it should; ansible-playbook task of template-validator failed. Expected results: ssp-operator should consume up to 3m cpu; playbook should complete successfully on all tasks. Additional info: