Bugzilla will be upgraded to version 5.0. The upgrade date is tentatively scheduled for 2 December 2018, pending final testing and feedback.
Bug 1435485 - [Pool] VMs are still created with duplicate MAC addresses after 4.0.7 upgrade
[Pool] VMs are still created with duplicate MAC addresses after 4.0.7 upgrade
Status: CLOSED ERRATA
Product: Red Hat Enterprise Virtualization Manager
Classification: Red Hat
Component: ovirt-engine (Show other bugs)
4.0.7
x86_64 Linux
high Severity urgent
: ovirt-4.2.0
: ---
Assigned To: Dan Kenigsberg
Michael Burman
: ZStream
Depends On: 1492723
Blocks: 1485688
  Show dependency treegraph
 
Reported: 2017-03-23 19:46 EDT by Germano Veit Michel
Modified: 2018-05-15 13:42 EDT (History)
19 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Previously, when creating large pools of virtual machines, the Engine assigned duplicate MAC addresses, causing network problems. In the current release, the problem has been solved by removing a problematic legacy method.
Story Points: ---
Clone Of:
: 1485688 (view as bug list)
Environment:
Last Closed: 2018-05-15 13:41:09 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: Network
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 2775081 None None None 2017-03-23 20:47 EDT
oVirt gerrit 76309 master POST core: removing 'berserk' forceAddMac method 2017-08-15 12:01 EDT
oVirt gerrit 78132 master MERGED core: fixes & simplifications in MoveMacs.java 2017-08-09 03:49 EDT
oVirt gerrit 78146 master MERGED core: removal of addForce method from cluster and DC update. 2017-08-09 08:39 EDT
oVirt gerrit 78191 master MERGED core: simplified parameters passed to VmInterfaceManager#add 2017-08-09 08:39 EDT
oVirt gerrit 78192 master POST core: remove forceAddMac from VmInterfaceManager,ImportVmCommandBase 2017-08-16 16:09 EDT
oVirt gerrit 78193 master ABANDONED core: add validation for duplicate macs when importing vm 2017-07-10 10:28 EDT
oVirt gerrit 80450 master POST core: when initializing MacPool also register in it nics in snapshots 2017-08-16 16:09 EDT
oVirt gerrit 81477 master MERGED core: removed duplicity in code 2017-09-18 08:38 EDT
oVirt gerrit 83415 master MERGED packaging: Introduce helper to fix duplicate macs. 2018-03-25 08:55 EDT
oVirt gerrit 89524 master MERGED examples: Introduce example to fix duplicate macs. 2018-04-03 09:42 EDT
oVirt gerrit 89766 sdk_4.2 MERGED examples: Introduce example to fix duplicate macs. 2018-04-03 09:47 EDT
Red Hat Product Errata RHEA-2018:1488 None None None 2018-05-15 13:42 EDT

  None (edit)
Description Germano Veit Michel 2017-03-23 19:46:12 EDT
Description of problem:

We have a report of the following BZ not being fixed by it's 4.0.7 clone:
BZ1400043 - [Vm Pool] VMs are created with duplicate MAC addresses

First try of new version (4.0.7) resulted in 12 VMs with Duplicate MACs.

Version-Release number of selected component (if applicable):
rhevm-4.0.7.4-0.1.el7ev.noarch
Comment 6 Germano Veit Michel 2017-03-24 01:44:22 EDT
Upgrade to 4.0.7 was from 4.0.6
Comment 7 Germano Veit Michel 2017-03-27 03:06:02 EDT
One interesting thing is that they have 5-6 VM Pools. Could this increase the probability of hitting the bug? It seems very easy to hit in that environment.
Comment 9 Alona Kaplan 2017-04-05 12:06:01 EDT
A possible reproduction of the bug -

-Make sure the MacPool used by the dc doesn't allow duplicates.

1. Create a template with one vnic ('tmp1').
2. Create VmPool ('pool') from 'tmp1' with 2 vms ('pool-1' and 'pool-2'). Set the number of prestarted vms as 2.
3. Wait for the vms to be up.
4. Unplug the nic from vm 'pool-1' (lets call its current mac address 'x'). Change its mac address (new mac 'y'). Plug it back.
5. Add a vnic to vm 'pool-2' and set its mac address to 'x' (the old mac address of the vnic we uplugged and plugged).
6. Stop vm 'pool-1'.

Result - Both vms 'pool-1' and 'pool-2' have vnic with 'x' mac.


Explanation of what causes the bug - when stopping a vm that was started by the pool, the original snapshot (before the run) is restored. The macs of the vnics in the original snapshot are added to the mac pool using 'forceAdd'. It means that it ignores if the mac is already in the pool.
So if a mac in the original snapshot was taken by another vm. We will end up with duplicate macs.
Comment 10 Germano Veit Michel 2017-04-23 21:44:15 EDT
Latest logs after a new test (with the snapshot related errors fixed) do not show the problem anymore.

I believe we are hitting the scenario Alona described, as the MAC Pool was close to been exhausted therefore the chances of another VM taking the MAC of the original snapshot were quite high.
Comment 25 Michael Burman 2017-09-18 10:13:33 EDT
This is failed QA, the result is the same a was befor ehte fix.
We don't reserve the origin MAC address for stateless VM and we can end up with duplicate MAC addresses because of that when the stateless VM is shutdown.
Comment 26 Dan Kenigsberg 2017-09-27 09:52:30 EDT
Rumor has it that currently merged code is fitting for QA.
Comment 27 Meni Yakove 2017-09-28 04:34:45 EDT
ovirt-engine-4.2.0-0.0.master.20170926175518.git0d20200.el7.centos.noarch
Comment 28 Michael Burman 2017-09-28 04:36:53 EDT
Verified on - 4.2.0-0.0.master.20170927183005.git49790b2.el7.centos
Comment 29 Michael Burman 2017-09-28 04:39:32 EDT
Summary and results:

Stateless scenarios - PASS
Statefull/snapshot scenarios - PASS
Regression - All new regression bugs which has been caused by the fix for this report has been verified
Tier 2 - PASS
MAC pool per cluster - no regression in the feature
Comment 33 RHV Bugzilla Automation and Verification Bot 2017-12-06 11:16:40 EST
INFO: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Open patch attached]

For more info please contact: rhv-devops@redhat.com
Comment 34 RHV Bugzilla Automation and Verification Bot 2017-12-12 16:15:09 EST
INFO: Bug status wasn't changed from MODIFIED to ON_QA due to the following reason:

[Open patch attached]

For more info please contact: rhv-devops@redhat.com
Comment 35 RHV Bugzilla Automation and Verification Bot 2017-12-18 12:05:33 EST
INFO: Bug status (VERIFIED) wasn't changed but the folowing should be fixed:

[Open patch attached]

For more info please contact: rhv-devops@redhat.com
Comment 37 Sandro Bonazzola 2018-04-18 07:17:47 EDT
Dan, this bug is verified but has an open patch attached: either drop the attached patch or move back this bug to POST.
Comment 38 Dan Kenigsberg 2018-04-24 09:58:21 EDT
This BZ has been (ab)used for examples on how to work around it. They have been merged 3 weeks ago, so it is not clear to me which patches you refer to.
Comment 41 errata-xmlrpc 2018-05-15 13:41:09 EDT
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:1488

Note You need to log in before you can comment on or make changes to this bug.