Bug 2210070 - During draining node non-migratable VM may block other VMs from being migrated for a long time
Summary: During draining node non-migratable VM may block other VMs from being migrate...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Container Native Virtualization (CNV)
Classification: Red Hat
Component: Virtualization
Version: 4.13.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.14.0
Assignee: Igor Bezukh
QA Contact: Kedar Bidarkar
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-05-25 16:22 UTC by Denys Shchedrivyi
Modified: 2023-11-08 14:06 UTC (History)
1 user (show)

Fixed In Version: hco-bundle-v4.14.0.rhel9-1854
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-11-08 14:05:46 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github kubevirt kubevirt pull 10349 0 None Draft [virt-controller] don't count pending migrations when validating the allowed migrations limit 2023-08-28 13:05:17 UTC
Red Hat Issue Tracker CNV-29308 0 None None None 2023-05-31 12:23:09 UTC
Red Hat Product Errata RHSA-2023:6817 0 None None None 2023-11-08 14:06:11 UTC

Description Denys Shchedrivyi 2023-05-25 16:22:07 UTC
Description of problem:
 By default we allow only 2 migrations in parallel for one node. If I have many non-migratable VMs on the cluster - they may block good VMs from being migrated during the drainig node.
 
 We have a fixed bug 2124528 about similar scenario - we added backoff to failed migrations and it helps with migrations which run manually (VMIM in Pending state does not affect on manually started migration). But the fix does not work well for automatically running migrations (i.e. kubevirt-evacuation and kubevirt-workload-update). 
 For example, I have multiple VMs running on one node:

> $ oc get vmi
> NAME                AGE   PHASE     IP             NODENAME                            READY
> vm-a                13h   Running   10.129.2.144   virt-den-413-zj5v7-worker-0-mr7cc   True
> vm-b                13h   Running   10.129.2.127   virt-den-413-zj5v7-worker-0-mr7cc   True
> vm-fedora-node1-1   13h   Running   10.129.2.115   virt-den-413-zj5v7-worker-0-mr7cc   True
> vm-fedora-node1-2   13h   Running   10.129.2.114   virt-den-413-zj5v7-worker-0-mr7cc   True
> vm-fedora-node1-3   13h   Running   10.129.2.116   virt-den-413-zj5v7-worker-0-mr7cc   True
> vm-fedora-node1-4   13h   Running   10.129.2.117   virt-den-413-zj5v7-worker-0-mr7cc   True
> vm-fedora-node1-5   13h   Running   10.129.2.119   virt-den-413-zj5v7-worker-0-mr7cc   True
> vm-fedora-node1-6   13h   Running   10.129.2.118   virt-den-413-zj5v7-worker-0-mr7cc   True
> vm-y                13h   Running   10.129.2.142   virt-den-413-zj5v7-worker-0-mr7cc   True
> vm-z                13h   Running   10.129.2.143   virt-den-413-zj5v7-worker-0-mr7cc   True

 vm-a(b,y,z) - migratable
 vm-fedora-node1-1(2,3...) - non-migratable because of node-selector

After draining the node if both VMIs selected for migration have some migration backoff - their VMIM stuck in Pending state and does not allow other VMIs to create migration:

> $ oc get vmim | egrep -v "Failed|Succeed"
> NAME                        PHASE       VMI
> kubevirt-evacuation-4s6td   Pending     vm-fedora-node1-3
> kubevirt-evacuation-77fr7   Pending     vm-fedora-node1-1

 ^^ VMIM in Pending state because of backoff, they doing nothing - just waiting when backoff timeout will expire. No any other VMIs are trying to migrate.

 But if I run migration manually - it succesfully created
> $ oc get vmim | egrep -v "Failed|Succeed"
> NAME                        PHASE       VMI
> kubevirt-evacuation-4s6td   Pending     vm-fedora-node1-3
> kubevirt-evacuation-77fr7   Pending     vm-fedora-node1-1
> kubevirt-evacuation-ftd25   Running     vm-z


Version-Release number of selected component (if applicable):
4.13

How reproducible:
 most of the time `good` VMs migrated in appropriate amount of time, but in rare cases I've observed upto 1hr for VM to be migrated.

Steps to Reproduce:
1. run multiple VMs (migratable and non-migratable) on one node
2. drain the node
3. 

Actual results:
 Migrations in Pending state (waiting when backoff timeout expored) does not allow other VMs to be migrated because of parallel migrations limits

Expected results:
  May be we should consider Pending migrations as non-active and they should not affect on the total number of parallel migratons during the node drain?

Additional info:

Comment 1 Denys Shchedrivyi 2023-09-22 00:53:03 UTC
Verified on v4.14.0.rhel9-2029

Pending migrations not counted as Active and don't affect on the total number of parallel migratons during the node drain


$ oc get vmim | grep -vE "Failed|Succeed"
NAME                        PHASE        VMI
kubevirt-evacuation-56kpk   Pending	 vm-fedora-node1-4
kubevirt-evacuation-jstkw   Scheduling	 vm-fedora-node1-2
kubevirt-evacuation-sszxl   Scheduling   vm-fedora-node1-1
kubevirt-evacuation-stfm6   Pending	 vm-fedora-node1-6
kubevirt-evacuation-svkdx   Pending   	 vm-fedora-node1-3
kubevirt-evacuation-sxz67   Pending      vm-fedora-node1-5

Comment 3 errata-xmlrpc 2023-11-08 14:05:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Virtualization 4.14.0 Images security and bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2023:6817


Note You need to log in before you can comment on or make changes to this bug.