Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: The virt-controllers are in CrashLoopBackOff state 5945 times (and counting) Version-Release number of selected component (if applicable): cnv 2.5.6 How reproducible: Always Steps to Reproduce: 1. uncertain... 2. 3. Actual results: virt-controller is in CrashLoopBackOff state and users can't launch new VMs Expected results: virt-controller "just works" Additional info: here are the logs from one of the failing controllers: [dyocum@dyocum ibm-cnv]$ omg logs virt-controller-f795c65f-cl4h5 /home/dyocum/Downloads/ibm-cnv/inspect.local.8117919182570694323/namespaces/openshift-cnv/pods/virt-controller-f795c65f-cl4h5/virt-controller/virt-controller/logs/current.log 2021-06-10T21:59:18.039718483Z 2021/06/10 21:59:18 WARNING: proto: file "api.proto" is already registered 2021-06-10T21:59:18.039718483Z A future release will panic on registration conflicts. See: 2021-06-10T21:59:18.039718483Z https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict 2021-06-10T21:59:18.039718483Z 2021-06-10T21:59:18.039827120Z 2021/06/10 21:59:18 WARNING: proto: file "api.proto" is already registered 2021-06-10T21:59:18.039827120Z A future release will panic on registration conflicts. See: 2021-06-10T21:59:18.039827120Z https://developers.google.com/protocol-buffers/docs/reference/go/faq#namespace-conflict 2021-06-10T21:59:18.039827120Z 2021-06-10T21:59:18.042464998Z {"component":"virt-controller","level":"info","msg":"STARTING informer configMapInformer","pos":"virtinformers.go:251","timestamp":"2021-06-10T21:59:18.042428Z"} 2021-06-10T21:59:18.042495991Z {"component":"virt-controller","level":"info","msg":"STARTING informer CRDInformer","pos":"virtinformers.go:251","timestamp":"2021-06-10T21:59:18.042468Z"} 2021-06-10T21:59:18.042523199Z {"component":"virt-controller","level":"info","msg":"STARTING informer kubeVirtInformer","pos":"virtinformers.go:251","timestamp":"2021-06-10T21:59:18.042496Z"} 2021-06-10T21:59:18.343198512Z {"component":"virt-controller","level":"info","msg":"Updating cluster config to resource version '272974767'","pos":"config-map.go:488","timestamp":"2021-06-10T21:59:18.343072Z"} 2021-06-10T21:59:18.343474281Z {"component":"virt-controller","level":"info","msg":"CDI detected, DataVolume integration enabled","pos":"application.go:273","timestamp":"2021-06-10T21:59:18.343348Z"} 2021-06-10T21:59:18.344235020Z {"action":"listening","component":"virt-controller","interface":"0.0.0.0","level":"info","port":8443,"pos":"application.go:322","service":"http","timestamp":"2021-06-10T21:59:18.344199Z"} 2021-06-10T21:59:18.344490986Z {"component":"virt-controller","level":"info","msg":"certificate with common name 'virt-controller.openshift-cnv.pod.cluster.local' retrieved.","pos":"cert-manager.go:196","service":"http","timestamp":"2021-06-10T21:59:18.344466Z"} 2021-06-10T21:59:34.582371424Z {"component":"virt-controller","level":"info","msg":"SKIPPING informer kubeVirtInformer","pos":"virtinformers.go:248","service":"http","timestamp":"2021-06-10T21:59:34.582260Z"} 2021-06-10T21:59:34.582371424Z {"component":"virt-controller","level":"info","msg":"STARTING informer vmInformer","pos":"virtinformers.go:251","service":"http","timestamp":"2021-06-10T21:59:34.582292Z"} 2021-06-10T21:59:34.582371424Z {"component":"virt-controller","level":"info","msg":"STARTING informer kubeVirtPodInformer","pos":"virtinformers.go:251","service":"http","timestamp":"2021-06-10T21:59:34.582307Z"} 2021-06-10T21:59:34.582371424Z {"component":"virt-controller","level":"info","msg":"STARTING informer storageClassInformer","pos":"virtinformers.go:251","service":"http","timestamp":"2021-06-10T21:59:34.582334Z"} 2021-06-10T21:59:34.582371424Z {"component":"virt-controller","level":"info","msg":"STARTING informer dataVolumeInformer","pos":"virtinformers.go:251","service":"http","timestamp":"2021-06-10T21:59:34.582343Z"} 2021-06-10T21:59:34.582371424Z {"component":"virt-controller","level":"info","msg":"STARTING informer vmimInformer","pos":"virtinformers.go:251","service":"http","timestamp":"2021-06-10T21:59:34.582353Z"} 2021-06-10T21:59:34.582399430Z {"component":"virt-controller","level":"info","msg":"STARTING informer vmSnapshotContentInformer","pos":"virtinformers.go:251","service":"http","timestamp":"2021-06-10T21:59:34.582363Z"} 2021-06-10T21:59:34.582399430Z {"component":"virt-controller","level":"info","msg":"STARTING informer vmRestoreInformer","pos":"virtinformers.go:251","service":"http","timestamp":"2021-06-10T21:59:34.582384Z"} 2021-06-10T21:59:34.582430221Z {"component":"virt-controller","level":"info","msg":"STARTING informer podInformer","pos":"virtinformers.go:251","service":"http","timestamp":"2021-06-10T21:59:34.582394Z"} 2021-06-10T21:59:34.582430221Z {"component":"virt-controller","level":"info","msg":"STARTING informer vmSnapshotInformer","pos":"virtinformers.go:251","service":"http","timestamp":"2021-06-10T21:59:34.582403Z"} 2021-06-10T21:59:34.582430221Z {"component":"virt-controller","level":"info","msg":"SKIPPING informer configMapInformer","pos":"virtinformers.go:248","service":"http","timestamp":"2021-06-10T21:59:34.582414Z"} 2021-06-10T21:59:34.582437176Z {"component":"virt-controller","level":"info","msg":"SKIPPING informer CRDInformer","pos":"virtinformers.go:248","service":"http","timestamp":"2021-06-10T21:59:34.582423Z"} 2021-06-10T21:59:34.582443445Z {"component":"virt-controller","level":"info","msg":"STARTING informer vmiInformer","pos":"virtinformers.go:251","service":"http","timestamp":"2021-06-10T21:59:34.582432Z"} 2021-06-10T21:59:34.582459648Z {"component":"virt-controller","level":"info","msg":"STARTING informer kubeVirtNodeInformer","pos":"virtinformers.go:251","service":"http","timestamp":"2021-06-10T21:59:34.582442Z"} 2021-06-10T21:59:34.582465890Z {"component":"virt-controller","level":"info","msg":"STARTING informer vmirsInformer","pos":"virtinformers.go:251","service":"http","timestamp":"2021-06-10T21:59:34.582455Z"} 2021-06-10T21:59:34.582479441Z {"component":"virt-controller","level":"info","msg":"STARTING informer persistentVolumeClaimInformer","pos":"virtinformers.go:251","service":"http","timestamp":"2021-06-10T21:59:34.582464Z"} 2021-06-10T21:59:34.582503797Z {"component":"virt-controller","level":"info","msg":"STARTING controllers with following threads : node 3, vmi 3, replicaset 3, vm 3, migration 3, evacuation 3, disruptionBudget 3\n","pos":"application.go:364","timestamp":"2021-06-10T21:59:34.582488Z"} 2021-06-10T21:59:34.582544432Z {"component":"virt-controller","level":"info","msg":"Starting evacuation controller.","pos":"evacuation.go:254","service":"http","timestamp":"2021-06-10T21:59:34.582528Z"} 2021-06-10T21:59:34.582582397Z {"component":"virt-controller","level":"info","msg":"Starting migration controller.","pos":"migration.go:112","service":"http","timestamp":"2021-06-10T21:59:34.582565Z"} 2021-06-10T21:59:34.582713679Z {"component":"virt-controller","level":"info","msg":"Starting VirtualMachine controller.","pos":"vm.go:121","service":"http","timestamp":"2021-06-10T21:59:34.582566Z"} 2021-06-10T21:59:34.582713679Z {"component":"virt-controller","level":"info","msg":"Starting snapshot controller.","pos":"snapshot_base.go:150","service":"http","timestamp":"2021-06-10T21:59:34.582598Z"} 2021-06-10T21:59:34.582713679Z {"component":"virt-controller","level":"info","msg":"Starting restore controller.","pos":"restore_base.go:88","service":"http","timestamp":"2021-06-10T21:59:34.582622Z"} 2021-06-10T21:59:34.582713679Z {"component":"virt-controller","level":"info","msg":"Starting VirtualMachineInstanceReplicaSet controller.","pos":"replicaset.go:110","service":"http","timestamp":"2021-06-10T21:59:34.582635Z"} 2021-06-10T21:59:34.582713679Z {"component":"virt-controller","level":"info","msg":"Starting node controller.","pos":"node.go:109","service":"http","timestamp":"2021-06-10T21:59:34.582616Z"} 2021-06-10T21:59:34.582713679Z {"component":"virt-controller","level":"info","msg":"Starting vmi controller.","pos":"vmi.go:176","service":"http","timestamp":"2021-06-10T21:59:34.582682Z"} 2021-06-10T21:59:34.582739394Z {"component":"virt-controller","level":"info","msg":"Starting disruption budget controller.","pos":"disruptionbudget.go:252","service":"http","timestamp":"2021-06-10T21:59:34.582672Z"} 2021-06-10T21:59:34.603612057Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"poc2-disk-0-y7otc","namespace":"smd-vm-group","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.603537Z","uid":"e33fbd81-52b8-42de-be96-732b5d9aeb8f"} 2021-06-10T21:59:34.603637763Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: poc2-disk-0-y7otc","name":"poc2-disk-0-y7otc","namespace":"smd-vm-group","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.603593Z","uid":"e33fbd81-52b8-42de-be96-732b5d9aeb8f"} 2021-06-10T21:59:34.603637763Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"test-rhel-disk-0-o8p2h","namespace":"smd-vm-group","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.603615Z","uid":"5f84cfef-8c4c-4086-939f-987ca26020e5"} 2021-06-10T21:59:34.603657917Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: test-rhel-disk-0-o8p2h","name":"test-rhel-disk-0-o8p2h","namespace":"smd-vm-group","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.603633Z","uid":"5f84cfef-8c4c-4086-939f-987ca26020e5"} 2021-06-10T21:59:34.603683461Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"icabi-disk-0-z15iu","namespace":"coe-test","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.603664Z","uid":"f2cd3702-4228-4e87-9507-9cd74b2fdf0c"} 2021-06-10T21:59:34.603704164Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: icabi-disk-0-z15iu","name":"icabi-disk-0-z15iu","namespace":"coe-test","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.603680Z","uid":"f2cd3702-4228-4e87-9507-9cd74b2fdf0c"} 2021-06-10T21:59:34.603727207Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"kostest2-disk-1-pyhyp","namespace":"coe-test","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.603705Z","uid":"bb2c058d-1c3f-466d-b408-4ddd619a2403"} 2021-06-10T21:59:34.603746830Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: kostest2-disk-1-pyhyp","name":"kostest2-disk-1-pyhyp","namespace":"coe-test","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.603725Z","uid":"bb2c058d-1c3f-466d-b408-4ddd619a2403"} 2021-06-10T21:59:34.603767375Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"sherwin-disk-0-ecjrk","namespace":"joel-cnv","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.603747Z","uid":"df946898-df3c-48ba-b173-3ccaf5a2753b"} 2021-06-10T21:59:34.603787025Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: sherwin-disk-0-ecjrk","name":"sherwin-disk-0-ecjrk","namespace":"joel-cnv","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.603765Z","uid":"df946898-df3c-48ba-b173-3ccaf5a2753b"} 2021-06-10T21:59:34.603809795Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"with-storage-blockdisk-azj0a","namespace":"rob-virt","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.603789Z","uid":"bc82a878-cb1f-436f-8845-0d464c40999f"} 2021-06-10T21:59:34.603829142Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: with-storage-blockdisk-azj0a","name":"with-storage-blockdisk-azj0a","namespace":"rob-virt","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.603808Z","uid":"bc82a878-cb1f-436f-8845-0d464c40999f"} 2021-06-10T21:59:34.603851311Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"test-disk-0-eqwze","namespace":"ciosreimages","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.603829Z","uid":"e0a07642-61b2-407c-842c-66557fd8f1df"} 2021-06-10T21:59:34.603878231Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: test-disk-0-eqwze","name":"test-disk-0-eqwze","namespace":"ciosreimages","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.603849Z","uid":"e0a07642-61b2-407c-842c-66557fd8f1df"} 2021-06-10T21:59:34.603911341Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"kosnewpvctest-disk-0-xhtt8","namespace":"coe-test","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.603882Z","uid":"a099e7c6-63b5-4444-9917-05211ebfbac2"} 2021-06-10T21:59:34.603931316Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: kosnewpvctest-disk-0-xhtt8","name":"kosnewpvctest-disk-0-xhtt8","namespace":"coe-test","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.603909Z","uid":"a099e7c6-63b5-4444-9917-05211ebfbac2"} 2021-06-10T21:59:34.603952480Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"persistent-disk-0","namespace":"joel-cnv","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.603932Z","uid":"5cd15c18-82cd-4096-a4c5-30a232cfea84"} 2021-06-10T21:59:34.603972104Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: persistent-disk-0","name":"persistent-disk-0","namespace":"joel-cnv","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.603950Z","uid":"5cd15c18-82cd-4096-a4c5-30a232cfea84"} 2021-06-10T21:59:34.603994687Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"bluemine-disk-0-h6u9o","namespace":"bluemine","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.603974Z","uid":"1a5f1f69-4631-429c-8ad8-20d03f92cdff"} 2021-06-10T21:59:34.604013596Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: bluemine-disk-0-h6u9o","name":"bluemine-disk-0-h6u9o","namespace":"bluemine","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.603993Z","uid":"1a5f1f69-4631-429c-8ad8-20d03f92cdff"} 2021-06-10T21:59:34.604034015Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"kosnewpvctest-disk-1-0q79g","namespace":"coe-test","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.604014Z","uid":"0e8ce856-9299-4220-ae8f-bdf74528ec79"} 2021-06-10T21:59:34.604053902Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: kosnewpvctest-disk-1-0q79g","name":"kosnewpvctest-disk-1-0q79g","namespace":"coe-test","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.604032Z","uid":"0e8ce856-9299-4220-ae8f-bdf74528ec79"} 2021-06-10T21:59:34.604076373Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"dcsoxyauthlic-disk-0-d9vyx","namespace":"dcs-openshift-virtualization","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.604057Z","uid":"058e55d8-5833-4c52-b9a0-d9773e07c4d9"} 2021-06-10T21:59:34.604098377Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: dcsoxyauthlic-disk-0-d9vyx","name":"dcsoxyauthlic-disk-0-d9vyx","namespace":"dcs-openshift-virtualization","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.604076Z","uid":"058e55d8-5833-4c52-b9a0-d9773e07c4d9"} 2021-06-10T21:59:34.604120261Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"8333-disk-333-gvwsl","namespace":"joel-cnv","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.604098Z","uid":"be0da6c2-115e-4362-bc70-cfd2aa3e848b"} 2021-06-10T21:59:34.604140546Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: 8333-disk-333-gvwsl","name":"8333-disk-333-gvwsl","namespace":"joel-cnv","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.604119Z","uid":"be0da6c2-115e-4362-bc70-cfd2aa3e848b"} 2021-06-10T21:59:34.604170974Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"cnv-image-update-1-disk-0-4hacy","namespace":"cnv-image-test","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.604145Z","uid":"60a72506-4cac-4922-b300-e12b26eb8165"} 2021-06-10T21:59:34.604196094Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: cnv-image-update-1-disk-0-4hacy","name":"cnv-image-update-1-disk-0-4hacy","namespace":"cnv-image-test","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.604177Z","uid":"60a72506-4cac-4922-b300-e12b26eb8165"} 2021-06-10T21:59:34.604219992Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"cnv-image-update-1-disk-1-qiruw","namespace":"cnv-image-test","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.604197Z","uid":"54dd6971-53a2-4069-9d65-efc18387b12f"} 2021-06-10T21:59:34.604241150Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: cnv-image-update-1-disk-1-qiruw","name":"cnv-image-update-1-disk-1-qiruw","namespace":"cnv-image-test","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.604219Z","uid":"54dd6971-53a2-4069-9d65-efc18387b12f"} 2021-06-10T21:59:34.604262751Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"kostest2-disk-0-zy2kf","namespace":"coe-test","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.604243Z","uid":"ee392af8-7ea2-4114-9dfc-9f4c7dd4cb4c"} 2021-06-10T21:59:34.604282720Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: kostest2-disk-0-zy2kf","name":"kostest2-disk-0-zy2kf","namespace":"coe-test","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.604262Z","uid":"ee392af8-7ea2-4114-9dfc-9f4c7dd4cb4c"} 2021-06-10T21:59:34.604304221Z {"component":"virt-controller","kind":"DataVolume","level":"info","msg":"Looking for DataVolume Ref","name":"redthor-rootdisk-hasvb","namespace":"gvat-cirrus","pos":"vm.go:988","service":"http","timestamp":"2021-06-10T21:59:34.604283Z","uid":"2eb24596-dd78-4b12-a6d6-8b6216c480af"} 2021-06-10T21:59:34.604325438Z {"component":"virt-controller","kind":"DataVolume","level":"error","msg":"Cant find the matching VM for DataVolume: redthor-rootdisk-hasvb","name":"redthor-rootdisk-hasvb","namespace":"gvat-cirrus","pos":"vm.go:991","service":"http","timestamp":"2021-06-10T21:59:34.604302Z","uid":"2eb24596-dd78-4b12-a6d6-8b6216c480af"} 2021-06-10T21:59:34.682945429Z E0610 21:59:34.682881 1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference) 2021-06-10T21:59:34.682945429Z goroutine 617 [running]: 2021-06-10T21:59:34.682945429Z k8s.io/apimachinery/pkg/util/runtime.logPanic(0x19f7d00, 0x2f1cd60) 2021-06-10T21:59:34.682945429Z /go/src/kubevirt.io/kubevirt/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0xa3 2021-06-10T21:59:34.682945429Z k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) 2021-06-10T21:59:34.682945429Z /go/src/kubevirt.io/kubevirt/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x82 2021-06-10T21:59:34.682945429Z panic(0x19f7d00, 0x2f1cd60) 2021-06-10T21:59:34.682945429Z /usr/lib/golang/src/runtime/panic.go:679 +0x1b2 2021-06-10T21:59:34.682945429Z kubevirt.io/kubevirt/pkg/virt-controller/watch.(*MigrationController).sync(0xc00182cbd0, 0xc003d96ac0, 0x36, 0xc0004eebd0, 0xc002656dc8, 0x2f7a5b8, 0x0, 0x0, 0x1c9947a, 0x3f) 2021-06-10T21:59:34.682945429Z /go/src/kubevirt.io/kubevirt/pkg/virt-controller/watch/migration.go:546 +0x1b4 2021-06-10T21:59:34.682945429Z kubevirt.io/kubevirt/pkg/virt-controller/watch.(*MigrationController).execute(0xc00182cbd0, 0xc003d96ac0, 0x36, 0x0, 0x60) 2021-06-10T21:59:34.682945429Z /go/src/kubevirt.io/kubevirt/pkg/virt-controller/watch/migration.go:215 +0x948 2021-06-10T21:59:34.682945429Z kubevirt.io/kubevirt/pkg/virt-controller/watch.(*MigrationController).Execute(0xc00182cbd0, 0xc00381d400) 2021-06-10T21:59:34.682945429Z /go/src/kubevirt.io/kubevirt/pkg/virt-controller/watch/migration.go:137 +0x109 2021-06-10T21:59:34.682945429Z kubevirt.io/kubevirt/pkg/virt-controller/watch.(*MigrationController).runWorker(0xc00182cbd0) 2021-06-10T21:59:34.682945429Z /go/src/kubevirt.io/kubevirt/pkg/virt-controller/watch/migration.go:127 +0x2b 2021-06-10T21:59:34.682945429Z k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0021e8890) 2021-06-10T21:59:34.682945429Z /go/src/kubevirt.io/kubevirt/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:152 +0x5e 2021-06-10T21:59:34.682945429Z k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0021e8890, 0x3b9aca00, 0x0, 0x1, 0xc0009c3860) 2021-06-10T21:59:34.682945429Z /go/src/kubevirt.io/kubevirt/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:153 +0xf8 2021-06-10T21:59:34.682945429Z k8s.io/apimachinery/pkg/util/wait.Until(0xc0021e8890, 0x3b9aca00, 0xc0009c3860) 2021-06-10T21:59:34.682945429Z /go/src/kubevirt.io/kubevirt/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d 2021-06-10T21:59:34.682945429Z created by kubevirt.io/kubevirt/pkg/virt-controller/watch.(*MigrationController).Run 2021-06-10T21:59:34.682945429Z /go/src/kubevirt.io/kubevirt/pkg/virt-controller/watch/migration.go:119 +0x38a 2021-06-10T21:59:34.686479143Z panic: runtime error: invalid memory address or nil pointer dereference [recovered] 2021-06-10T21:59:34.686479143Z panic: runtime error: invalid memory address or nil pointer dereference 2021-06-10T21:59:34.686479143Z [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x180f6b4] 2021-06-10T21:59:34.686479143Z 2021-06-10T21:59:34.686479143Z goroutine 617 [running]: 2021-06-10T21:59:34.686479143Z k8s.io/apimachinery/pkg/util/runtime.HandleCrash(0x0, 0x0, 0x0) 2021-06-10T21:59:34.686479143Z /go/src/kubevirt.io/kubevirt/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0x1052021-06-10T21:59:34.686534959Z 2021-06-10T21:59:34.686534959Z panic(0x19f7d00, 0x2f1cd60) 2021-06-10T21:59:34.686534959Z /usr/lib/golang/src/runtime/panic.go:679 +0x1b2 2021-06-10T21:59:34.686534959Z kubevirt.io/kubevirt/pkg/virt-controller/watch.(*MigrationController).sync(0xc00182cbd0, 0xc003d96ac0, 0x36, 0xc0004eebd0, 0xc002656dc8, 0x2f7a5b8, 2021-06-10T21:59:34.686558507Z 0x0, 0x0, 0x1c9947a, 0x3f) 2021-06-10T21:59:34.686558507Z /go/src/kubevirt.io/kubevirt/pkg/virt-controller/watch/migration.go:546 +2021-06-10T21:59:34.686577192Z 0x1b4 2021-06-10T21:59:34.686577192Z kubevirt.io/kubevirt/pkg/virt-controller/watch.(*MigrationController).execute(0xc00182cbd0, 0xc003d96ac02021-06-10T21:59:34.686608144Z , 0x36, 0x0, 0x60) 2021-06-10T21:59:34.686608144Z /go/src/kubevirt.io/kubevirt/pkg/virt-controller/watch/migration.go:215 +0x948 2021-06-10T21:59:34.686608144Z kubevirt.io/kubevirt/pkg/virt-controller/watch.(*MigrationController).Execute(2021-06-10T21:59:34.686628608Z 0xc00182cbd0, 0xc00381d400) 2021-06-10T21:59:34.686628608Z /go/src/kubevirt.io/kubevirt/pkg/virt-controller/watch/migration.go:137 +0x109 2021-06-10T21:59:34.686646256Z kubevirt.io/kubevirt/pkg/virt-controller/watch.(*MigrationController).runWorker(0xc00182cbd0) 2021-06-10T21:59:34.686646256Z /go/src/kubevirt.io/kubevirt/pkg/virt-controller/watch/migration.go:127 +0x2b2021-06-10T21:59:34.686663896Z 2021-06-10T21:59:34.686663896Z k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1(0xc0021e8890) 2021-06-10T21:59:34.686663896Z /go/src/kubevirt.io/kubevirt/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:1522021-06-10T21:59:34.686682700Z +0x5e 2021-06-10T21:59:34.686682700Z k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc0021e8890, 0x3b9aca00, 2021-06-10T21:59:34.686700156Z 0x0, 0x1, 0xc0009c3860) 2021-06-10T21:59:34.686700156Z /go/src/kubevirt.io/kubevirt/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:1532021-06-10T21:59:34.686717240Z +0xf8 2021-06-10T21:59:34.686717240Z k8s.io/apimachinery/pkg/util/wait.Until(0xc0021e8890, 0x3b9aca00, 2021-06-10T21:59:34.686734472Z 0xc0009c3860) 2021-06-10T21:59:34.686734472Z /go/src/kubevirt.io/kubevirt/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88 +0x4d 2021-06-10T21:59:34.686752068Z created by kubevirt.io/kubevirt/pkg/virt-controller/watch.(*MigrationController).Run 2021-06-10T21:59:34.686752068Z /go/src/kubevirt.io/kubevirt/pkg/virt-controller/watch/migration.go:119 +2021-06-10T21:59:34.686769365Z 0x38a
related PR: https://github.com/kubevirt/kubevirt/pull/5707
verify with build: hco: 2.5.7-49 virt-operator-container-v2.5.6-8 step: 1. start migration 2. immediately delete the target pod right as it appears 3. check virt-controller status no crash loop occurs. check migration is failed. vm still running on source node. Virt-controller in running status do live migration again, it works. move to verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Virtualization 2.5.7 Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2021:2934