Bug 1583009

Hi,
I'll start with VM selection. We indeed sort the VMs according to cpu usage, with a preference to choosing the idle VMs first. The reason for this is that we prefer not to disturb a "busy" VM, while migrating an idle VM is more likely to succeed with no service disruption. So this is working by design and should stay this way.

As for the affinity rules; All hard affinity rules have been preserved, as they should. Soft affinity, by design, is best effort and may be disrupted when needed. So an an overall the system is functioning without breaking any rules. In this case one might claim that leaving the busy VM alone to avoid disruption is the right way to go. The alternative, of migrating a busy VM with a potential for migration failure and service disruption is the worst case scenario for some of the users who need the stability. As you can see there's not a single solution which fits everyone here. For this reason you can generate your own cluster policy and change the weights as you see fit.

Please let me know if you need anything else.

Hi Doron,

I understand the point behind migrating the VM with lowest CPU usage, it does make sense. Thanks for that.

However, it doesn't change much the erratic behavior. Let me explain it differently.

Scenario:
Host is overloaded
1 VM will lot of CPU usage
1 VM will medium CPU usage
3 VMs with no CPU usage, all part of a affinity group, pinned to host etc. Anything that ties them to the running host either by filtering or best score.

The current algorithm:

if host is overcommited:
   vms = migratable_vms(host)
   sort(vms, cpu_usage)
   for vm in vms:
       dest_hosts = getValidHosts(vm)
       if dest_hosts:
           return vm,dest_hosts

1) VM is always the VM with the lowest CPU usage. 
2) getValidHosts is a simple check. Passing it does not mean the VM can migrate.
3) For several reasons (hard pinning (filter), soft pinning (weight)..) the selected VM will be found to no be able to migrate (or the running host already has the best score).

Then the algorithm stops and no VMs are migrated. Then it runs again 1 minute later and hits the same problem, no VMs are migrated ever. The host continues overloaded and the VMs are providing poor service.

This loop here:

for (VM vmToMigrate : migratableVmsOnHost){                                                                                               
    // Check if vm not over utilize memory or CPU of destination hosts                                                                    
    List<VDS> validDestinationHosts = getValidHosts(                                                                                      
        destinationHosts, cluster, vmToMigrate, highCpuUtilization, requiredMemory);                                                  
                                                                                                                                                      
    if (!validDestinationHosts.isEmpty()){                                                                                                
        log.debug("Vm '{}' selected for migration", vmToMigrate.getName());                                                               
        return Optional.of(new Result(vmToMigrate, validDestinationHosts));                                                               
    }                                                                                                                                     
}

It will always return a VM that cannot migrate on every cycle. The scheduler is stuck, doing nothing.

IMO, if the VM cannot migrate, then the loop must continue and select the next VM with lowest CPU usage on the same cycle. Eventually it will find a VM that can migrate, even if it is the last VM. So it does not get stuck on these scenarios where the VM with lowest CPU usage will not migrate due to filtering or weights.

What do you think?

Hi Germano,

you are right, the current balancing algorithm is pretty simple and does not know how to handle this kind of situation. Count in the AREM that runs as a separate "balancing" thread and has similar limitations.

The issue here is that the balancing would have to check for every candidate VM if it has a viable target that actually improves anything [1] and that is a pretty big change. We would have to merge AREM and balancing and run a complete scheduling run from balancing without executing the results just to select the candidates. And then trigger the scheduler again to actually do something.

I was experimenting with a deterministic solution to this myself and I have a solution for this. Unfortunately, it is CPU intensive - balancing run for 10000 VMs took about 2 seconds to compute the result.

We rejected the other option (some kind of randomization) for support purposes. We would not be able to reproduce bug reports.

This can all be implemented and improved, but what we has been "good enough" so far and we could not justify the time and resources.


[1] The scheduler can decide that the best host for a VM is the current one.

Hi Martin,

> I was experimenting with a deterministic solution to this myself and I have
> a solution for this. Unfortunately, it is CPU intensive - balancing run for
> 10000 VMs took about 2 seconds to compute the result.

Interesting. Well, with 10k VMs I believe there will be other problems as well ;)
And with 10k VMs, its expected that RHV-M will have plenty of resources. 2s for 10k VMs doesn't sound that bad to me.

> This can all be implemented and improved, but what we has been "good enough"
> so far and we could not justify the time and resources.

There is a ticket attached to this BZ, so perhaps this could be considered a RFE?

Yep, we can consider this to be an RFE.

how about just adding a "last migration failure" time as primary sort-by field?

(In reply to Michal Skrivanek from comment #8)
> how about just adding a "last migration failure" time as primary sort-by
> field?

Or keep the sorted list the way it is. And then retry a migration with the next VM in the list if the migration validation fails.

Is this too intrusive for 4.2.z or we have plans to Z-Stream it once the patches are merged? Note there are tickets attached.

Probably too intrusive for a Z-stream unless there's an urgent need

WARN: Bug status (ON_QA) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{'rhevm-4.3-ga': '?'}', ]

For more info please contact: rhv-devops: Bug status (ON_QA) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{'rhevm-4.3-ga': '?'}', ]

For more info please contact: rhv-devops

WARN: Bug status (ON_QA) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{'rhevm-4.3-ga': '?'}', ]

For more info please contact: rhv-devops: Bug status (ON_QA) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{'rhevm-4.3-ga': '?'}', ]

For more info please contact: rhv-devops

WARN: Bug status (ON_QA) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{}', ]

For more info please contact: rhv-devops: Bug status (ON_QA) wasn't changed but the folowing should be fixed:

[Found non-acked flags: '{}', ]

For more info please contact: rhv-devops

Verified on:
ovirt-engine-4.3.2.1-0.0.master.20190305140204.git3649df7.el7.noarch

Steps:
Tested the scenario from comment #2.

Environment is with 3 hosts, 32cpus, SPM and HE VM were on host1.

1. Run 3 VMs with 1vcpu under hard affinity to host1.
2. Run VM-VM1 with mid load (24vcpus), causing a ~30% of the host cpu load on host1.
3. Run VM-VM2 with high load (28vcpus), causing a ~60% of the host cpu load on host1.
- total cpu load on host1 > 90%
4. Set even distribute scheduling policy on the cluster, with high utilization: 80%.

Results:
HE VM(without any load) migrated away to host2.
Afterwards, VM1 migrated away.

Another test was a negative test:
On the same environment as above. HE VM now on host2.
1. Run 3 VMs with 1vcpu under hard affinity to host1, load 1VM to 100% (total of ~3% of the host).
2. Run VM-VM1 with mid load (24vcpus), causing a ~30% of the host cpu load on host2.
3. Run VM-VM2 with high load (28vcpus), causing a ~60% of the host cpu load on host1.
- total cpu load on host1 > 63%
4. Set even distribute scheduling policy on the cluster, with high utilization: 55%.

Results:
As expected, host1 has > 63% load so it needs balance. host2 already have ~30%, moving VM2 to it is impossible, as the 3VMs(because affinity).
host3 is free but moving VM2 to it will cause a need to balance again.
The engine keeps trying to balance the cluster, each time he is trying different VM.
The engine does not invoke migration.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:1085

The needinfo request[s] on this closed bug have been removed as they have been unresolved for 120 days