Bug 1719190
| Summary: | Unable to cancel live-migration if virt-launcher pod in pending state | ||
|---|---|---|---|
| Product: | Container Native Virtualization (CNV) | Reporter: | Asher Shoshan <ashoshan> |
| Component: | Virtualization | Assignee: | Itamar Holder <iholder> |
| Status: | CLOSED ERRATA | QA Contact: | Akriti Gupta <akrgupta> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 2.0 | CC: | acardace, awax, dshchedr, fdeutsch, gouyang, kbidarka, sgott, vromanso |
| Target Milestone: | --- | ||
| Target Release: | 4.12.0 | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | hco-bundle-registry-container-v4.12.0-308 virt-handler-4.12.0-110 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-01-24 13:36:05 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Asher Shoshan
2019-06-11 09:32:42 UTC
*** Bug 1812775 has been marked as a duplicate of this bug. *** I did some digging into this, as well as talked it over with Vladik. This is a bit tricky to solve, but it might be possible. The issue we'd like to avoid is a race condition. The basic (successful) workflow is this: 1) a VMI is running 2) a VMI migration object is created 3) migration controller schedules a pod on a target node 4) once that pod is running, the migration controller will coordinate handoff between the virt-handlers of the respective nodes The breakdown here is with step 3. The controller has no way of differentiating between a pod that can't be scheduled due to cluster constraints (e.g. no nodes available) and a pod that is in the process of being scheduled and is about to run. The scenario that we'd like to avoid is to have a zombie virt-launcher pod that's running on a node, but no entity is responsible for it. This could happen if e.g. the migration object were deleted as the pod was being scheduled--there would be no VMI representing the pod, so virt-handler on that node would not recognize it. Currently KubeVirt has a finalizer that forbids migrations be deleted if they're not in a final state. This normally works fine, but in a case like the scenario in this BZ, the migration can't enter a final state because the destination pod is waiting to be scheduled. While we work on the best path forward, there does exist a workaround. Delete the target virt-launcher pod. the VMI Migration object will immediately enter the "failed" state. This meets the criterion of the finalizer, and the cluster will allow the migration object to be deleted. In order to ensure you delete the correct pod, there is an activePods stanza in the VMI's status. This is a mapping of pod's UID to node. the one with an empty node is the UID of interest (since that's the one not scheduled). The pod with the name "virt-launcher-XXXX" that is in a pending state with that UID is the target pod. Because there exists a workaround, and there is no danger of data loss with this workflow, I'm lowering the severity of this BZ to medium. *** Bug 2035924 has been marked as a duplicate of this bug. *** Since 2020 a lot has changed. If we still see migrations hanging when the new target pod is in pending state, we should be able properly abort nowadays. Let's check this out again. Properly setting the abort request on the VMI by the migration controller should give the source virt-handler and virt-controller all the info needed to abort in this stage in a race-free way. It can of course still happen that a migration can't be aborted anymore and still succeeds, but that is logically fine. verified with cnv v4.12.0-548, virt-handler v4.12.0-201 [akrgupta@fedora ~]$ oc get vmi NAME AGE PHASE IP NODENAME READY vm-example 29s Running 10.128.2.177 virt-den-412-pqfpv-worker-0-mjg69 True [akrgupta@fedora ~]$ oc adm cordon virt-den-412-pqfpv-worker-0-bg7n4 virt-den-412-pqfpv-worker-0-4tgwh node/virt-den-412-pqfpv-worker-0-bg7n4 cordoned node/virt-den-412-pqfpv-worker-0-4tgwh cordoned [akrgupta@fedora ~]$ virtctl migrate vm-example VM vm-example was scheduled to migrate [akrgupta@fedora ~]$ oc get vmim NAME PHASE VMI kubevirt-migrate-vm-n4dc2 Scheduling vm-example [akrgupta@fedora ~]$ oc get pods NAME READY STATUS RESTARTS AGE virt-launcher-vm-example-dp55v 0/2 Pending 0 102s virt-launcher-vm-example-wwznb 2/2 Running 0 6m41s [akrgupta@fedora ~]$ oc get vmim NAME PHASE VMI kubevirt-migrate-vm-n4dc2 Scheduling vm-example [akrgupta@fedora ~]$ oc delete vmim kubevirt-migrate-vm-n4dc2 virtualmachineinstancemigration.kubevirt.io "kubevirt-migrate-vm-n4dc2" deleted [akrgupta@fedora ~]$ oc get vmim No resources found in default namespace. [akrgupta@fedora ~]$ oc get vmi NAME AGE PHASE IP NODENAME READY vm-example 10m Running 10.128.2.177 virt-den-412-pqfpv-worker-0-mjg69 True able to cancel live-migration if virt-launcher pod in pending state , by deleting the vmim object Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: OpenShift Virtualization 4.12.0 Images security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2023:0408 |