Bug 1971007

Summary: virt-controller panics, enters CrashLoopBackOff state repeatedly
Product: Container Native Virtualization (CNV) Reporter: Dan Yocum <dyocum>
Component: VirtualizationAssignee: sgott
Status: CLOSED ERRATA QA Contact: Israel Pinto <ipinto>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 2.5.6CC: cnv-qe-bugs, dvossel, zpeng
Target Milestone: ---   
Target Release: 2.5.7   
Hardware: All   
OS: Linux   
Whiteboard:
Fixed In Version: virt-operator-container-v2.5.6-8 hco-bundle-registry-container-v2.5.7-27 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-07-28 11:29:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Dan Yocum 2021-06-11 16:08:42 UTC 
Description of problem:
The virt-controllers are in CrashLoopBackOff state 5945 times (and counting)

Version-Release number of selected component (if applicable):
cnv 2.5.6


How reproducible:
Always

Steps to Reproduce:
1. uncertain...
2.
3.

Actual results:

virt-controller is in CrashLoopBackOff state and users can't launch new VMs


Expected results:

virt-controller "just works"


Additional info:

here are the logs from one of the failing controllers:

[dyocum@dyocum ibm-cnv]$ omg logs virt-controller-f795c65f-cl4h5
/home/dyocum/Downloads/ibm-cnv/inspect.local.8117919182570694323/namespaces/openshift-cnv/pods/virt-controller-f795c65f-cl4h5/virt-controller/virt-controller/logs/current.log
2021-06-10T21:59:18.039718483Z 2021/06/10 21:59:18 WARNING: proto: file "api.proto" is already registered
2021-06-10T21:59:18.039718483Z A future release will panic on registration conflicts. See:
2021-06-10T21:59:18.039718483Z https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict
2021-06-10T21:59:18.039718483Z 
2021-06-10T21:59:18.039827120Z 2021/06/10 21:59:18 WARNING: proto: file "api.proto" is already registered
2021-06-10T21:59:18.039827120Z A future release will panic on registration conflicts. See:
2021-06-10T21:59:18.039827120Z https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict
2021-06-10T21:59:18.039827120Z 
2021-06-10T21:59:18.042464998Z {"component":"virt-controller","level":"info","msg":"STARTING informer configMapInformer","pos":"virtinformers.go:251","timestamp":"2021-06-10T21:59:18.042428Z"}
2021-06-10T21:59:18.042495991Z {"component":"virt-controller","level":"info","msg":"STARTING informer CRDInformer","pos":"virtinformers.go:251","timestamp":"2021-06-10T21:59:18.042468Z"}
2021-06-10T21:59:18.042523199Z {"component":"virt-controller","level":"info","msg":"STARTING informer kubeVirtInformer","pos":"virtinformers.go:251","timestamp":"2021-06-10T21:59:18.042496Z"}
2021-06-10T21:59:18.343198512Z {"component":"virt-controller","level":"info","msg":"Updating cluster config to resource version '272974767'","pos":"config-map.go:488","timestamp":"2021-06-10T21:59:18.343072Z"}
2021-06-10T21:59:18.343474281Z {"component":"virt-controller","level":"info","msg":"CDI detected, DataVolume integration enabled","pos":"application.go:273","timestamp":"2021-06-10T21:59:18.343348Z"}
2021-06-10T21:59:18.344235020Z {"action":"listening","component":"virt-controller","interface":"0.0.0.0","level":"info","port":8443,"pos":"application.go:322","service":"http","timestamp":"2021-06-10T21:59:18.344199Z"}
2021-06-10T21:59:18.344490986Z {"component":"virt-controller","level":"info","msg":"certificate with common name 'virt-controller.openshift-cnv.pod.cluster.local' retrieved.","pos":"cert-manager.go:196","service":"http","timestamp":"2021-06-10T21:59:18.344466Z"}
2021-06-10T21:59:34.582371424Z {"component":"virt-controller","level":"info","msg":"SKIPPING informer kubeVirtInformer","pos":"virtinformers.go:248","service":"http","timestamp":"2021-06-10T21:59:34.582260Z"}
2021-06-10T21:59:34.582371424Z {"component":"virt-controller","level":"info","msg":"STARTING informer vmInformer","pos":"virtinformers.go:251","service":"http","timestamp":"2021-06-10T21:59:34.582292Z"}
2021-06-10T21:59:34.582371424Z {"component":"virt-controller","level":"info","msg":"STARTING informer kubeVirtPodInformer","pos":"virtinformers.go:251","service":"http","timestamp":"2021-06-10T21:59:34.582307Z"}
2021-06-10T21:59:34.582371424Z {"component":"virt-controller","level":"info","msg":"STARTING informer storageClassInformer","pos":"virtinformers.go:251","service":"http","timestamp":"2021-06-10T21:59:34.582334Z"}
2021-06-10T21:59:34.582371424Z {"component":"virt-controller","level":"info","msg":"STARTING informer dataVolumeInformer","pos":"virtinformers.go:251","service":"http","timestamp":"2021-06-10T21:59:34.582343Z"}
2021-06-10T21:59:34.582371424Z {"component":"virt-controller","level":"info","msg":"STARTING informer vmimInformer","pos":"virtinformers.go:251","service":"http","timestamp":"2021-06-10T21:59:34.582353Z"}
2021-06-10T21:59:34.582399430Z {"component":"virt-controller","level":"info","msg":"STARTING informer vmSnapshotContentInformer","pos":"virtinformers.go:251","service":"http","timestamp":"2021-06-10T21:59:34.582363Z"}
2021-06-10T21:59:34.582399430Z {"component":"virt-controller","level":"info","msg":"STARTING informer vmRestoreInformer","pos":"virtinformers.go:251","service":"http","timestamp":"2021-06-10T21:59:34.582384Z"}
2021-06-10T21:59:34.582430221Z {"component":"virt-controller","level":"info","msg":"STARTING informer podInformer","pos":"virtinformers.go:251","service":"http","timestamp":"2021-06-10T21:59:34.582394Z"}
2021-06-10T21:59:34.582430221Z {"component":"virt-controller","level":"info","msg":"STARTING informer vmSnapshotInformer","pos":"virtinformers.go:251","service":"http","timestamp":"2021-06-10T21:59:34.582403Z"}
2021-06-10T21:59:34.582430221Z {"component":"virt-controller","level":"info","msg":"SKIPPING informer configMapInformer","pos":"virtinformers.go:248","service":"http","timestamp":"2021-06-10T21:59:34.582414Z"}
2021-06-10T21:59:34.582437176Z {"component":"virt-controller","level":"info","msg":"SKIPPING informer CRDInformer","pos":"virtinformers.go:248","service":"http","timestamp":"2021-06-10T21:59:34.582423Z"}
2021-06-10T21:59:34.582443445Z {"component":"virt-controller","level":"info","msg":"STARTING informer vmiInformer","pos":"virtinformers.go:251","service":"http","timestamp":"2021-06-10T21:59:34.582432Z"}
2021-06-10T21:59:34.582459648Z {"component":"virt-controller","level":"info","msg":"STARTING informer kubeVirtNodeInformer","pos":"virtinformers.go:251","service":"http","timestamp":"2021-06-10T21:59:34.582442Z"}
2021-06-10T21:59:34.582465890Z {"component":"virt-controller","level":"info","msg":"STARTING informer vmirsInformer","pos":"virtinformers.go:251","service":"http","timestamp":"2021-06-10T21:59:34.582455Z"}
2021-06-10T21:59:34.582479441Z {"component":"virt-controller","level":"info","msg":"STARTING informer persistentVolumeClaimInformer","pos":"virtinformers.go:251","service":"http","timestamp":"2021-06-10T21:59:34.582464Z"}
2021-06-10T21:59:34.582503797Z {"component":"virt-controller","level":"info","msg":"STARTING controllers with following threads : node 3, vmi 3, replicaset 3, vm 3, migration 3, evacuation 3, disruptionBudget 3\n","pos":"application.go:364","timestamp":"2021-06-10T21:59:34.582488Z"}
2021-06-10T21:59:34.582544432Z {"component":"virt-controller","level":"info","msg":"Starting evacuation controller.","pos":"evacuation.go:254","service":"http","timestamp":"2021-06-10T21:59:34.582528Z"}
2021-06-10T21:59:34.582582397Z {"component":"virt-controller","level":"info","msg":"Starting migration controller.","pos":"migration.go:112","service":"http","timestamp":"2021-06-10T21:59:34.582565Z"}
2021-06-10T21:59:34.582713679Z {"component":"virt-controller","level":"info","msg":"Starting VirtualMachine controller.","pos":"vm.go:121","service":"http","timestamp":"2021-06-10T21:59:34.582566Z"}
2021-06-10T21:59:34.582713679Z {"component":"virt-controller","level":"info","msg":"Starting snapshot controller.","pos":"snapshot_base.go:150","service":"http","timestamp":"2021-06-10T21:59:34.582598Z"}
2021-06-10T21:59:34.582713679Z {"component":"virt-controller","level":"info","msg":"Starting restore controller.","pos":"restore_base.go:88","service":"http","timestamp":"2021-06-10T21:59:34.582622Z"}
2021-06-10T21:59:34.582713679Z {"component":"virt-controller","level":"info","msg":"Starting VirtualMachineInstanceReplicaSet controller.","pos":"replicaset.go:110","service":"http","timestamp":"2021-06-10T21:59:34.582635Z"}
2021-06-10T21:59:34.582713679Z {"component":"virt-controller","level":"info","msg":"Starting node controller.","pos":"node.go:109","service":"http","timestamp":"2021-06-10T21:59:34.582616Z"}
2021-06-10T21:59:34.582713679Z {"component":"virt-controller","level":"info","msg":"Starting vmi controller.","pos":"vmi.go:176","service":"http","timestamp":"2021-06-10T21:59:34.582682Z"}
2021-06-10T21:59:34.582739394Z {"component":"virt-controller","level":"info","msg":"Starting disruption budget controller.","pos":"disruptionbudget.go:252","service":"http","timestamp":"2021-06-10T21:59:34.582672Z"}
2021-06-10T21:59:34.603612057Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"poc2-disk-0-y7otc","namespace":"smd-vm-group","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.603537Z","uid":"e33fbd81-52b8-42de-be96-732b5d9aeb8f"}
2021-06-10T21:59:34.603637763Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: poc2-disk-0-y7otc","name":"poc2-disk-0-y7otc","namespace":"smd-vm-group","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.603593Z","uid":"e33fbd81-52b8-42de-be96-732b5d9aeb8f"}
2021-06-10T21:59:34.603637763Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"test-rhel-disk-0-o8p2h","namespace":"smd-vm-group","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.603615Z","uid":"5f84cfef-8c4c-4086-939f-987ca26020e5"}
2021-06-10T21:59:34.603657917Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: test-rhel-disk-0-o8p2h","name":"test-rhel-disk-0-o8p2h","namespace":"smd-vm-group","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.603633Z","uid":"5f84cfef-8c4c-4086-939f-987ca26020e5"}
2021-06-10T21:59:34.603683461Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"icabi-disk-0-z15iu","namespace":"coe-test","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.603664Z","uid":"f2cd3702-4228-4e87-9507-9cd74b2fdf0c"}
2021-06-10T21:59:34.603704164Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: icabi-disk-0-z15iu","name":"icabi-disk-0-z15iu","namespace":"coe-test","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.603680Z","uid":"f2cd3702-4228-4e87-9507-9cd74b2fdf0c"}
2021-06-10T21:59:34.603727207Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"kostest2-disk-1-pyhyp","namespace":"coe-test","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.603705Z","uid":"bb2c058d-1c3f-466d-b408-4ddd619a2403"}
2021-06-10T21:59:34.603746830Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: kostest2-disk-1-pyhyp","name":"kostest2-disk-1-pyhyp","namespace":"coe-test","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.603725Z","uid":"bb2c058d-1c3f-466d-b408-4ddd619a2403"}
2021-06-10T21:59:34.603767375Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"sherwin-disk-0-ecjrk","namespace":"joel-cnv","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.603747Z","uid":"df946898-df3c-48ba-b173-3ccaf5a2753b"}
2021-06-10T21:59:34.603787025Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: sherwin-disk-0-ecjrk","name":"sherwin-disk-0-ecjrk","namespace":"joel-cnv","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.603765Z","uid":"df946898-df3c-48ba-b173-3ccaf5a2753b"}
2021-06-10T21:59:34.603809795Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"with-storage-blockdisk-azj0a","namespace":"rob-virt","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.603789Z","uid":"bc82a878-cb1f-436f-8845-0d464c40999f"}
2021-06-10T21:59:34.603829142Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: with-storage-blockdisk-azj0a","name":"with-storage-blockdisk-azj0a","namespace":"rob-virt","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.603808Z","uid":"bc82a878-cb1f-436f-8845-0d464c40999f"}
2021-06-10T21:59:34.603851311Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"test-disk-0-eqwze","namespace":"ciosreimages","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.603829Z","uid":"e0a07642-61b2-407c-842c-66557fd8f1df"}
2021-06-10T21:59:34.603878231Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: test-disk-0-eqwze","name":"test-disk-0-eqwze","namespace":"ciosreimages","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.603849Z","uid":"e0a07642-61b2-407c-842c-66557fd8f1df"}
2021-06-10T21:59:34.603911341Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"kosnewpvctest-disk-0-xhtt8","namespace":"coe-test","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.603882Z","uid":"a099e7c6-63b5-4444-9917-05211ebfbac2"}
2021-06-10T21:59:34.603931316Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: kosnewpvctest-disk-0-xhtt8","name":"kosnewpvctest-disk-0-xhtt8","namespace":"coe-test","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.603909Z","uid":"a099e7c6-63b5-4444-9917-05211ebfbac2"}
2021-06-10T21:59:34.603952480Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"persistent-disk-0","namespace":"joel-cnv","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.603932Z","uid":"5cd15c18-82cd-4096-a4c5-30a232cfea84"}
2021-06-10T21:59:34.603972104Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: persistent-disk-0","name":"persistent-disk-0","namespace":"joel-cnv","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.603950Z","uid":"5cd15c18-82cd-4096-a4c5-30a232cfea84"}
2021-06-10T21:59:34.603994687Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"bluemine-disk-0-h6u9o","namespace":"bluemine","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.603974Z","uid":"1a5f1f69-4631-429c-8ad8-20d03f92cdff"}
2021-06-10T21:59:34.604013596Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: bluemine-disk-0-h6u9o","name":"bluemine-disk-0-h6u9o","namespace":"bluemine","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.603993Z","uid":"1a5f1f69-4631-429c-8ad8-20d03f92cdff"}
2021-06-10T21:59:34.604034015Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"kosnewpvctest-disk-1-0q79g","namespace":"coe-test","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.604014Z","uid":"0e8ce856-9299-4220-ae8f-bdf74528ec79"}
2021-06-10T21:59:34.604053902Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: kosnewpvctest-disk-1-0q79g","name":"kosnewpvctest-disk-1-0q79g","namespace":"coe-test","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.604032Z","uid":"0e8ce856-9299-4220-ae8f-bdf74528ec79"}
2021-06-10T21:59:34.604076373Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"dcsoxyauthlic-disk-0-d9vyx","namespace":"dcs-openshift-virtualization","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.604057Z","uid":"058e55d8-5833-4c52-b9a0-d9773e07c4d9"}
2021-06-10T21:59:34.604098377Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: dcsoxyauthlic-disk-0-d9vyx","name":"dcsoxyauthlic-disk-0-d9vyx","namespace":"dcs-openshift-virtualization","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.604076Z","uid":"058e55d8-5833-4c52-b9a0-d9773e07c4d9"}
2021-06-10T21:59:34.604120261Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"8333-disk-333-gvwsl","namespace":"joel-cnv","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.604098Z","uid":"be0da6c2-115e-4362-bc70-cfd2aa3e848b"}
2021-06-10T21:59:34.604140546Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: 8333-disk-333-gvwsl","name":"8333-disk-333-gvwsl","namespace":"joel-cnv","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.604119Z","uid":"be0da6c2-115e-4362-bc70-cfd2aa3e848b"}
2021-06-10T21:59:34.604170974Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"cnv-image-update-1-disk-0-4hacy","namespace":"cnv-image-test","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.604145Z","uid":"60a72506-4cac-4922-b300-e12b26eb8165"}
2021-06-10T21:59:34.604196094Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: cnv-image-update-1-disk-0-4hacy","name":"cnv-image-update-1-disk-0-4hacy","namespace":"cnv-image-test","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.604177Z","uid":"60a72506-4cac-4922-b300-e12b26eb8165"}
2021-06-10T21:59:34.604219992Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"cnv-image-update-1-disk-1-qiruw","namespace":"cnv-image-test","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.604197Z","uid":"54dd6971-53a2-4069-9d65-efc18387b12f"}
2021-06-10T21:59:34.604241150Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: cnv-image-update-1-disk-1-qiruw","name":"cnv-image-update-1-disk-1-qiruw","namespace":"cnv-image-test","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.604219Z","uid":"54dd6971-53a2-4069-9d65-efc18387b12f"}
2021-06-10T21:59:34.604262751Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"kostest2-disk-0-zy2kf","namespace":"coe-test","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.604243Z","uid":"ee392af8-7ea2-4114-9dfc-9f4c7dd4cb4c"}
2021-06-10T21:59:34.604282720Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: kostest2-disk-0-zy2kf","name":"kostest2-disk-0-zy2kf","namespace":"coe-test","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.604262Z","uid":"ee392af8-7ea2-4114-9dfc-9f4c7dd4cb4c"}
2021-06-10T21:59:34.604304221Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"redthor-rootdisk-hasvb","namespace":"gvat-cirrus","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.604283Z","uid":"2eb24596-dd78-4b12-a6d6-8b6216c480af"}
2021-06-10T21:59:34.604325438Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: redthor-rootdisk-hasvb","name":"redthor-rootdisk-hasvb","namespace":"gvat-cirrus","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.604302Z","uid":"2eb24596-dd78-4b12-a6d6-8b6216c480af"}
2021-06-10T21:59:34.682945429Z E0610 21:59:34.682881       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
2021-06-10T21:59:34.682945429Z goroutine 617 [running]:
2021-06-10T21:59:34.682945429Z k8s.io/apimachinery/pkg/util/runtime.logPanic(0x19f7d00, 0x2f1cd60)
2021-06-10T21:59:34.682945429Z 	/go/src/kubevirt.io/kubevirt/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa3
2021-06-10T21:59:34.682945429Z k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
2021-06-10T21:59:34.682945429Z 	/go/src/kubevirt.io/kubevirt/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x82
2021-06-10T21:59:34.682945429Z panic(0x19f7d00, 0x2f1cd60)
2021-06-10T21:59:34.682945429Z 	/usr/lib/golang/src/runtime/panic.go:679 +0x1b2
2021-06-10T21:59:34.682945429Z kubevirt.io/kubevirt/pkg/virt-controller/watch.(*MigrationController).sync(0xc00182cbd0, 0xc003d96ac0, 0x36, 0xc0004eebd0, 0xc002656dc8, 0x2f7a5b8, 0x0, 0x0, 0x1c9947a, 0x3f)
2021-06-10T21:59:34.682945429Z 	/go/src/kubevirt.io/kubevirt/pkg/virt-controller/watch/migration.go:546 +0x1b4
2021-06-10T21:59:34.682945429Z kubevirt.io/kubevirt/pkg/virt-controller/watch.(*MigrationController).execute(0xc00182cbd0, 0xc003d96ac0, 0x36, 0x0, 0x60)
2021-06-10T21:59:34.682945429Z 	/go/src/kubevirt.io/kubevirt/pkg/virt-controller/watch/migration.go:215 +0x948
2021-06-10T21:59:34.682945429Z kubevirt.io/kubevirt/pkg/virt-controller/watch.(*MigrationController).Execute(0xc00182cbd0, 0xc00381d400)
2021-06-10T21:59:34.682945429Z 	/go/src/kubevirt.io/kubevirt/pkg/virt-controller/watch/migration.go:137 +0x109
2021-06-10T21:59:34.682945429Z kubevirt.io/kubevirt/pkg/virt-controller/watch.(*MigrationController).runWorker(0xc00182cbd0)
2021-06-10T21:59:34.682945429Z 	/go/src/kubevirt.io/kubevirt/pkg/virt-controller/watch/migration.go:127 +0x2b
2021-06-10T21:59:34.682945429Z k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0021e8890)
2021-06-10T21:59:34.682945429Z 	/go/src/kubevirt.io/kubevirt/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x5e
2021-06-10T21:59:34.682945429Z k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0021e8890, 0x3b9aca00, 0x0, 0x1, 0xc0009c3860)
2021-06-10T21:59:34.682945429Z 	/go/src/kubevirt.io/kubevirt/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8
2021-06-10T21:59:34.682945429Z k8s.io/apimachinery/pkg/util/wait.Until(0xc0021e8890, 0x3b9aca00, 0xc0009c3860)
2021-06-10T21:59:34.682945429Z 	/go/src/kubevirt.io/kubevirt/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
2021-06-10T21:59:34.682945429Z created by kubevirt.io/kubevirt/pkg/virt-controller/watch.(*MigrationController).Run
2021-06-10T21:59:34.682945429Z 	/go/src/kubevirt.io/kubevirt/pkg/virt-controller/watch/migration.go:119 +0x38a
2021-06-10T21:59:34.686479143Z panic: runtime error: invalid memory address or nil pointer dereference [recovered]
2021-06-10T21:59:34.686479143Z 	panic: runtime error: invalid memory address or nil pointer dereference
2021-06-10T21:59:34.686479143Z [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x180f6b4]
2021-06-10T21:59:34.686479143Z 
2021-06-10T21:59:34.686479143Z goroutine 617 [running]:
2021-06-10T21:59:34.686479143Z k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0)
2021-06-10T21:59:34.686479143Z 	/go/src/kubevirt.io/kubevirt/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x1052021-06-10T21:59:34.686534959Z 
2021-06-10T21:59:34.686534959Z panic(0x19f7d00, 0x2f1cd60)
2021-06-10T21:59:34.686534959Z 	/usr/lib/golang/src/runtime/panic.go:679 +0x1b2
2021-06-10T21:59:34.686534959Z kubevirt.io/kubevirt/pkg/virt-controller/watch.(*MigrationController).sync(0xc00182cbd0, 0xc003d96ac0, 0x36, 0xc0004eebd0, 0xc002656dc8, 0x2f7a5b8, 2021-06-10T21:59:34.686558507Z 0x0, 0x0, 0x1c9947a, 0x3f)
2021-06-10T21:59:34.686558507Z 	/go/src/kubevirt.io/kubevirt/pkg/virt-controller/watch/migration.go:546 +2021-06-10T21:59:34.686577192Z 0x1b4
2021-06-10T21:59:34.686577192Z kubevirt.io/kubevirt/pkg/virt-controller/watch.(*MigrationController).execute(0xc00182cbd0, 0xc003d96ac02021-06-10T21:59:34.686608144Z , 0x36, 0x0, 0x60)
2021-06-10T21:59:34.686608144Z 	/go/src/kubevirt.io/kubevirt/pkg/virt-controller/watch/migration.go:215 +0x948
2021-06-10T21:59:34.686608144Z kubevirt.io/kubevirt/pkg/virt-controller/watch.(*MigrationController).Execute(2021-06-10T21:59:34.686628608Z 0xc00182cbd0, 0xc00381d400)
2021-06-10T21:59:34.686628608Z 	/go/src/kubevirt.io/kubevirt/pkg/virt-controller/watch/migration.go:137 +0x109
2021-06-10T21:59:34.686646256Z kubevirt.io/kubevirt/pkg/virt-controller/watch.(*MigrationController).runWorker(0xc00182cbd0)
2021-06-10T21:59:34.686646256Z 	/go/src/kubevirt.io/kubevirt/pkg/virt-controller/watch/migration.go:127 +0x2b2021-06-10T21:59:34.686663896Z 
2021-06-10T21:59:34.686663896Z k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0021e8890)
2021-06-10T21:59:34.686663896Z 	/go/src/kubevirt.io/kubevirt/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:1522021-06-10T21:59:34.686682700Z  +0x5e
2021-06-10T21:59:34.686682700Z k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0021e8890, 0x3b9aca00, 2021-06-10T21:59:34.686700156Z 0x0, 0x1, 0xc0009c3860)
2021-06-10T21:59:34.686700156Z 	/go/src/kubevirt.io/kubevirt/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:1532021-06-10T21:59:34.686717240Z  +0xf8
2021-06-10T21:59:34.686717240Z k8s.io/apimachinery/pkg/util/wait.Until(0xc0021e8890, 0x3b9aca00, 2021-06-10T21:59:34.686734472Z 0xc0009c3860)
2021-06-10T21:59:34.686734472Z 	/go/src/kubevirt.io/kubevirt/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d
2021-06-10T21:59:34.686752068Z created by kubevirt.io/kubevirt/pkg/virt-controller/watch.(*MigrationController).Run
2021-06-10T21:59:34.686752068Z 	/go/src/kubevirt.io/kubevirt/pkg/virt-controller/watch/migration.go:119 +2021-06-10T21:59:34.686769365Z 0x38a
Comment 5 David Vossel 2021-06-22 12:10:21 UTC 
related PR: https://github.com/kubevirt/kubevirt/pull/5707
Comment 6 zhe peng 2021-07-06 02:51:36 UTC 
verify with build:
hco: 2.5.7-49
virt-operator-container-v2.5.6-8

step:
1. start migration 
2. immediately delete the target pod right as it appears
3. check  virt-controller status

no crash loop occurs.
check migration is failed. vm still running on source node. Virt-controller in running status
do live migration again, it works. 

move to verified.
Comment 12 errata-xmlrpc 2021-07-28 11:29:38 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Virtualization 2.5.7 Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2021:2934