Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: machine-config-daemon was trying to evict the virt-launcher pod while draining the node. === pods/machine-config-daemon-2pqq7/machine-config-daemon/machine-config-daemon/logs/current.log 2021-04-02T16:51:23.102136850Z I0402 16:51:23.102017 9659 update.go:1676] Update prepared; beginning drain 2021-04-02T16:51:25.725425144Z I0402 16:51:25.725418 9659 daemon.go:344] evicting pod default/virt-launcher-test_vm-9kz5g 2021-04-02T16:51:35.798055503Z E0402 16:51:35.798012 9659 daemon.go:344] error when evicting pod "virt-launcher-test_vm-9kz5g" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget. === During the eviction process, the virt-controller tried to migrate the VM and it failed with the below error. === pods/virt-controller-5869d6fc64-xbk6k/virt-controller/virt-controller/logs/current.log 2021-04-02T16:52:22.318181410Z {"component":"virt-controller","kind":"","level":"error","msg":"target pod default/virt-launcher-test_vm-bvxgb shutdown during migration","name":"kubevirt-evacuation-ft7bx","namespace":"default","pos":"migration.go:286","service":"http","timestamp":"2021-04-02T16:52:22.318055Z","uid":"62ac0d50-311c-48a4-8bd2-562dfb443353"} === The virt-controller tried to migrate the VM again but this time, it failed with the error "Active migration job is currently already in progress for VMI". Here the VMI reporting "Migration State:" as "active" although it's already failed. === machineconfig]$ oc describe vmis test_vm Name: test_vm Namespace: default ........... .......... .......... Migration State: Migration UID: 62ac0d50-311c-48a4-8bd2-562dfb443353 Source Node: source.example.com Target Node: destination.example.com Target Pod: virt-launcher-test_vm-bvxgb Node Name: source.example.com Phase: Running Qos Class: Burstable === However, the vmim kubevirt-evacuation-ft7bx reflects correct status and marks migration as failed. === [kni@aospbas01 machineconfig]$ oc describe vmim kubevirt-evacuation-ft7bx Name: kubevirt-evacuation-ft7bx Namespace: default Labels: <none> Annotations: kubevirt.io/latest-observed-api-version: v1alpha3 kubevirt.io/storage-observed-api-version: v1alpha3 API Version: kubevirt.io/v1alpha3 Kind: VirtualMachineInstanceMigration Spec: Vmi Name: test_vm Status: Phase: Failed ======> Marked as failed. Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulCreate 21m virtualmachine-controller Created migration target pod virt-launcher-test_vm-bvxgb Normal SuccessfulHandOver 21m virtualmachine-controller Migration target pod is ready for preparation by virt-handler. Warning FailedMigration 21m virtualmachine-controller Migration failed because target pod shutdown during migration === I can also see that in the latest version, there are improvements in this area as per bug 1937920 and we are looking for vmim status now to confirm there is no existing active migration. https://github.com/kubevirt/kubevirt/pull/5242/files#diff-934c555aef2d898f826aa77568e99853e5e11dc0b3d1fb62a81c74ac768f0983R59 Version-Release number of selected component (if applicable): v2.5.5 How reproducible: Reproduced intermittently in the customer environment. Steps to Reproduce: The issue was observed during the node drain. Actual results: VM does not migrate after it fails to migrate at the first attempt during node drain
There were multiple attempts to migrate the VM but all failed with the same error. === 2021-04-02T16:53:53.516330822Z {"component":"virt-api","level":"error","msg":"admission generic error","pos":"webhooks.go:41","reason":"in-flight migration detected. Active migration job (62ac0d50-311c-48a4-8bd2-562dfb443353) is currently already in progress for VMI test_vm.","timestamp":"2021-04-02T16:53:53.516155Z"} 2021-04-02T17:51:16.921103413Z {"component":"virt-api","level":"error","msg":"admission generic error","pos":"webhooks.go:41","reason":"in-flight migration detected. Active migration job (62ac0d50-311c-48a4-8bd2-562dfb443353) is currently already in progress for VMI test_vm.","timestamp":"2021-04-02T17:51:16.920944Z"} === The customer was only able to drain the node after powering down the VM.
verify with build hco-bundle-registry-container-v2.5.6-65 virt-operator-container-v2.5.6-3 scenario 1: step: create a vm and start start migration start another migration job for same vm before first job end get error: Error from server: error when creating "migration-job.yaml": admission webhook "migration-create-validator.kubevirt.io" denied the request: in-flight migration detected. scenario 2: create 10 vms in one node do node drain, migration of VM started, wait until migration finished, check vm status, migration all finished w/o error. also test some negative cases, no issues found, move to verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Virtualization 2.5.6 Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2021:2045