Created attachment 1654301 [details] ssp-operator logs, from the beginning. Description of problem: ssp-operator is running an ansible-playbook that continuously crashing at the template-validator task, thus consuming high amount of cpu usage ~ 1.1 CPU core on average. Version-Release number of selected component (if applicable): SSP version: registry.redhat.io/container-native-virtualization/kubevirt-ssp-operator:v2.1.0-22 Openshift version: 4.4.0-0.ci-2020-01-19-121550 HCO version: registry.redhat.io/container-native-virtualization/hyperconverged-cluster-operator:v2.1.0-22 How reproducible: 100% Steps to Reproduce: 1. Deploy CNV 2.1 on Openshift 4.4 2. 3. Actual results: ssp-operator consumes three order of magnitude more CPU than it should; ansible-playbook task of template-validator failed. Expected results: ssp-operator should consume up to 3m cpu; playbook should complete successfully on all tasks. Additional info:
This PR will fix it: https://github.com/MarSik/kubevirt-ssp-operator/pull/142
Verified: OCP 4.4.0-rc.2 CNV 2.3 (v0.26.3) 1. SSP operator deploys template-validator ------------------------------------------ virt-template-validator-5bfb67ff94-6chct 1/1 Running 0 93m virt-template-validator-5bfb67ff94-qt54r 1/1 Running 0 86m 2. No errors in SSP oeprator log -------------------------------- {"level":"info","ts":1584625326.419964,"logger":"logging_event_handler","msg":"[playbook task]","name":"template-validator-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"kubevirt.io/v1, Kind=KubevirtTemplateValidator","event_type":"playbook_on_task_start","job":"2740103009342231109","EventData.Name":"Gathering Facts"} {"level":"info","ts":1584625331.1642094,"logger":"logging_event_handler","msg":"[playbook debug]","name":"template-validator-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"kubevirt.io/v1, Kind=KubevirtTemplateValidator","event_type":"runner_on_ok","job":"2740103009342231109","EventData.TaskArgs":""} {"level":"info","ts":1584625331.183627,"logger":"logging_event_handler","msg":"[playbook task]","name":"template-validator-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"kubevirt.io/v1, Kind=KubevirtTemplateValidator","event_type":"playbook_on_task_start","job":"2740103009342231109","EventData.Name":"KubevirtTemplateValidator : Set template:view role"} {"level":"info","ts":1584625332.967145,"logger":"logging_event_handler","msg":"[playbook task]","name":"template-validator-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"kubevirt.io/v1, Kind=KubevirtTemplateValidator","event_type":"playbook_on_task_start","job":"2740103009342231109","EventData.Name":"KubevirtTemplateValidator : Create the service"} {"level":"info","ts":1584625338.4236267,"logger":"logging_event_handler","msg":"[playbook task]","name":"template-validator-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"kubevirt.io/v1, Kind=KubevirtTemplateValidator","event_type":"playbook_on_task_start","job":"2740103009342231109","EventData.Name":"KubevirtTemplateValidator : Register the webhook"} {"level":"info","ts":1584625340.5451968,"logger":"logging_event_handler","msg":"[playbook task]","name":"template-validator-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"kubevirt.io/v1, Kind=KubevirtTemplateValidator","event_type":"playbook_on_task_start","job":"2740103009342231109","EventData.Name":"KubevirtTemplateValidator : Set progressing condition"} {"level":"info","ts":1584625342.7724156,"logger":"logging_event_handler","msg":"[playbook task]","name":"template-validator-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"kubevirt.io/v1, Kind=KubevirtTemplateValidator","event_type":"playbook_on_task_start","job":"2740103009342231109","EventData.Name":"KubevirtTemplateValidator : Wait for the template-validator to start"} {"level":"info","ts":1584625344.5061178,"logger":"logging_event_handler","msg":"[playbook task]","name":"template-validator-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"kubevirt.io/v1, Kind=KubevirtTemplateValidator","event_type":"playbook_on_task_start","job":"2740103009342231109","EventData.Name":"KubevirtTemplateValidator : Set available condition"} {"level":"info","ts":1584625345.9363396,"logger":"logging_event_handler","msg":"[playbook task]","name":"template-validator-kubevirt-hyperconverged","namespace":"openshift-cnv","gvk":"kubevirt.io/v1, Kind=KubevirtTemplateValidator","event_type":"playbook_on_task_start","job":"2740103009342231109","EventData.Name":"KubevirtTemplateValidator : Set degraded condition"} {"level":"info","ts":1584625347.7245126,"logger":"runner","msg":"Ansible-runner exited successfully","job":"2740103009342231109","name":"template-validator-kubevirt-hyperconverged","namespace":"openshift-cnv"} 3. Validate apiVersion in template-validator deplyment ------------------------------------------------------ $ oc get deployment -n openshift-cnv virt-template-validator -oyaml|grep -A2 app apiVersion: apps/v1 kind: Deployment 4. Validate template-validator pods' cpu: ----------------------------------------- $ for i in `oc get pod -n openshift-cnv -oname|grep valida|cut -d'/' -f2`; do echo $i; kubectl top pod $i --containers -n openshift-cnv; done virt-template-validator-5bfb67ff94-6chct POD NAME CPU(cores) MEMORY(bytes) virt-template-validator-5bfb67ff94-6chct webhook 1m 20Mi virt-template-validator-5bfb67ff94-qt54r POD NAME CPU(cores) MEMORY(bytes) virt-template-validator-5bfb67ff94-qt54r webhook 1m 16Mi
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:2011