Bug 1651747

Summary: All virtual machines in an affinity group do not run on same host after migrating one of the VMs with VM Soft Affinity.
Product: Red Hat Enterprise Virtualization Manager Reporter: Abhishekh Patil <abpatil>
Component: ovirt-engineAssignee: Andrej Krejcir <akrejcir>
Status: CLOSED ERRATA QA Contact: Polina <pagranat>
Severity: medium Docs Contact:
Priority: high    
Version: 4.2.7CC: akrejcir, amashah, avoss, cshao, emarcus, gchakkar, mavital, michal.skrivanek, mkalinin, rbarry, Rhev-m-bugs, rhodain, yquinn
Target Milestone: ovirt-4.3.5Flags: lsvaty: testing_plan_complete-
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: Enhancement
Doc Text:
In this release, the Affinity Enforcement process now includes Soft Virtual Machine Affinity.
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-08-12 11:53:27 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: Virt RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Description Flags
engine with debug none

Description Abhishekh Patil 2018-11-20 16:45:41 UTC
Description of problem:

Virtual machine does not migrate after enabling VM-to-VM soft Affinity (Positive VM Polarity). If we migrate one VM to other host, other VM in the affinity group does not migrate even after very long time.

Version-Release number of selected component (if applicable):


How reproducible:

Steps to Reproduce:

1) Create new Scheduling Policy for the Cluster (Administration -> Configure -> Scheduling Policies -> vm_evenly_distributed -> Copy (as Affinity_testing) -> Affinity_testing -> Edit:

Leave Filter Modules as-is, remove all but VmAffinityGroups Weight Module, set its Factor to 100, disable Load Balancer, leave Properties as-is.

2) Select the Policy for the Cluster from Compute -> Clusters -> Default -> Edit -> Scheduling Policy -> Affinity_testing and leave everything as defaults.

3) Create two VMs on Compute -> Virtual Machines and in the VM view check that both have under Affinity Groups an Affinity Group with VM Affinity Rule Positive (non-Enforcing) and Host Affinity Rule Disabled, both VMs listed under Virtual Machines.

4) Shutdown both VMs, start them both, notice how they start on the same Host. Migrate either of them to another Host and notice how the other VM does not follow it even after a long period of time.

Actual results:

VMs in an affinity group does not migrate even after migrating one VM from the affinity group to another host. 

Expected results:

The following documentation says "Soft enforcement - indicates a preference for virtual machines in an affinity group to run on the specified host or hosts in a group when possible." 


So, all VMs in an affinity group should run on same host or hosts in a group when possible.

Comment 1 Ryan Barry 2018-11-21 00:27:55 UTC
This still appears to be reproducible (I encountered it today)

Yanir, are there any changes to the default scheduling policy needed to make this work as expected? The policies I looked at today already had values for affinity set to priority 10

Comment 2 Andrej Krejcir 2018-11-21 10:42:45 UTC
Looking at the code, the component responsible for migrating VMs that break affinity rules (AffinityRulesEnforcementManager) does not consider soft VM-VM affinity when choosing a VM to migrate. I think this is a bug, AREM should try to correct soft VM-VM affinity too.

Currently the soft VM-VM affinity is only used when starting or migrating a VM and during balancing. But the balancing chooses which VM to migrate based on CPU and memory load, not affinity rules. So in case the hosts are not overloaded, the VMs will not be migrated even if they break soft affinity.

Comment 16 Gajanan 2019-02-20 11:09:56 UTC
Can someone help to check whether this bug is similar to this bug: Bug 1678708

Comment 17 Andrej Krejcir 2019-02-20 13:42:02 UTC
This bug is about enabling the automatic migration which fixes broken affinity groups even for soft VM affinity groups.
And Bug 1678708 looks to be about VM soft affinity not applying when starting VMs.

These bugs are different.

Comment 22 Polina 2019-06-23 12:58:38 UTC
Hi Andrej, 

I'm testing on ovirt-engine- according to the bz the patch must be there . But I see some unexpected behavior.

Sometimes I migrate the VM1 (placed in soft positive VMs affinity rule) and the VM2 is balanced as expected (because of the affinity rule) to the same new host to be together with VM1.
But in most cases I see another behavior:
I migrate the VM1 and wait for the VM2 to be balanced. Instead, though I see that the VM1 is balanced back to the source host.

like this :
Jun 23, 2019, 3:40:04 PM Migration started (VM: golden_env_mixed_virtio_0, Source: host_mixed_1, Destination: host_mixed_2, User: admin@internal-authz). 
Jun 23, 2019, 3:40:27 PM Migration completed (VM: golden_env_mixed_virtio_0, Source: host_mixed_1, Destination: host_mixed_2, Duration: 12 seconds, Total: 22 seconds, Actual downtime: 59ms)
Jun 23, 2019, 3:40:57 PM Migration initiated by system (VM: golden_env_mixed_virtio_0, Source: host_mixed_2, Destination: host_mixed_1, Reason: Affinity rules enforcement)
Jun 23, 2019, 3:41:12 PM Migration completed (VM: golden_env_mixed_virtio_0, Source: host_mixed_2, Destination: host_mixed_1, Duration: 4 seconds, Total: 14 seconds, Actual downtime: 63ms)

I attach engine log with scheduler debug . Please see example  starting from the line 
2019-06-23 15:40:04,502+03 INFO  [org.ovirt.engine.core.dal.dbbroker.auditloghandling.AuditLogDirector] (default task-3) [5a79e5ef-c2aa-409d-b606-c26b6f778088] EVENT_ID: VM_MIGRATION_START(62), Migration started (VM: golden_env_mixed_virtio_0, Source: host_mixed_1, Destination: host_mixed_2, User: admin@internal-authz).
Two VMs are running on host_mixed_1 . I migrate VM golden_env_mixed_virtio_0 to the host_mixed_2 . It succeeds and then after 30 sec this VM is balanced back instead of expected the second VM golden_env_mixed_virtio_1 to be migrated to the host host_mixed_2.

Could you please look?

Comment 23 Polina 2019-06-23 12:59:21 UTC
Created attachment 1583695 [details]
engine with debug

Comment 24 Andrej Krejcir 2019-06-24 07:34:46 UTC
That is the expected behavior. The affinity enforcement just tries to migrate VMs such that the affinity is not broken.
It does not matter if VM2 migrates to VM1 or VM1 migrates back to VM2, both cases are ok, since the affinity is fixed.

In case the user does not want VM1 to migrate back, its migration mode should be set to "manual migration only".

Comment 25 Polina 2019-06-24 10:52:08 UTC
verified on the base of https://bugzilla.redhat.com/show_bug.cgi?id=1651747#c22 and #c24

Comment 29 errata-xmlrpc 2019-08-12 11:53:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Comment 30 Daniel Gur 2019-08-28 13:12:32 UTC

Comment 31 Daniel Gur 2019-08-28 13:16:45 UTC