Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: given a VM is running on a worker when a migration job is created for the VM and the migration target worker is restarted then migration fails, and no additional attempts at migration are performed the virt-launcher pod, that was created on the target worker, remains in a failed state: status: message: Pod Predicate NodeAffinity failed phase: Failed reason: NodeAffinity startTime: "2021-07-14T13:59:59Z" the migration job remains in a failed state, and cannot be deleted, unless the finalizer is manually removed: Status: Phase: Failed Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SuccessfulCreate 33m virtualmachine-controller Created migration target pod virt-launcher-vm1-dmprq Warning FailedMigration 31m virtualmachine-controller Migration failed because target pod shutdown during migration the VMI keeps running on the initial node, reporting two active pods, one on the initial worker, the other on the target worker: status: activePods: 45ab65c3-f09a-477b-82ed-5dfd2aeb4261: worker-0-0 a95701e6-10a4-4e03-b403-32efc03c32a4: worker-0-2 Version-Release number of selected component (if applicable): v2.6.5 How reproducible: Consistently, using cnv-chaos Steps to Reproduce: 1. Start a vm in the cluster 2. Create a migration job for the VM 3. Reboot the migration target worker before migration finishes Actual results: VM migration fails, and no further attempts at migration are performed The VM keeps running on the initial worker Expected results: Upon failure, assuming it's a temporary one, additional attempts at migraiton are made, until migration is successful or a terminal failure occurs Additional info: This test was performed in two ways: 1. the VM runs on node A, node B is cordoned, VM is migrated to node C - this way the VM can only migrate to node C, which is being restarted 2. the VM runs on node A, VM is migrated to node C - this way, when node C is restarted, the VM can be migrated to node B instead, it fails nonetheless
When this condition is reached, how many VMI pods are listed for the VMI which are not in a "terminated" state? If the target pod scheduled on the node that's restarting never reaches a terminating state, then that will block any additional migrations from taking place until the previous target pod reaches "terminating"
(In reply to David Vossel from comment #1) > When this condition is reached, how many VMI pods are listed for the VMI > which are not in a "terminated" state? > > If the target pod scheduled on the node that's restarting never reaches a > terminating state, then that will block any additional migrations from > taking place until the previous target pod reaches "terminating" The target pod remains in "Failed" phase: [root@zeus19 scenario03]# oc get pods virt-launcher-vm1-tc9ln -o jsonpath='{.status}' | jq { "message": "Pod Predicate NodeAffinity failed", "phase": "Failed", "reason": "NodeAffinity", "startTime": "2021-07-14T16:18:52Z" } [root@zeus19 scenario03]# oc get pods virt-launcher-vm1-tc9ln -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES virt-launcher-vm1-tc9ln 0/2 NodeAffinity 0 12m <none> worker-0-1 <none> <none>
I think I missed a detail in the description. > given a VM is running on a worker when a migration job is created > for the VM and the migration target worker is restarted then > migration fails, and no additional attempts at migration are performed and > Upon failure, assuming it's a temporary one, additional attempts at migraiton are made, until migration is successful or a terminal failure occurs In this example, what is actually creating the Migration objects? If it's the test driver that's causing the migrations, then there's no expectation that a migration will be retried unless the test driver retries the live migration when it fails. Migration objects themselves are a one shot resource, similar to a Pod and a VMI. There's one attempt at a migration, and once that attempt fails or succeeds then that's the end of the migration's lifecycle. We have higher level controllers related to eviction and workload updating that will introduce migrations for VMIs and re-attempt failed migrations, but this behavior is built on top of the migration objects themselves. Based on the description, unless one of our higher level controllers is introducing the migrations, i would not expect a one off migration object to get retried if it fails.
Anton, Question per Comment #3, what is actually creating the Migration objects?
The migration was originally being created using a VMIM object. I have now retested this by draining the original node while the only available destination node is being rebooted. The behavior is as expected, migration completes post reboot.