Bug 2067216

Summary:	[EL9] add-port creates interfaces with the same MAC address on different nodes
Product:	Red Hat Enterprise Linux Fast Datapath	Reporter:	Cédric Jeanneret <cjeanner>
Component:	openvswitch2.15	Assignee:	Open vSwitch development team <ovs-team>
Status:	CLOSED NOTABUG	QA Contact:	ovs-qe
Severity:	urgent	Docs Contact:
Priority:	urgent
Version:	RHEL 9.0	CC:	apevec, ctrautma, jhsiao, ralongi
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-03-24 13:34:50 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Cédric Jeanneret 2022-03-23 14:44:50 UTC

Description of problem:
While testing EL9 and OSP-17, we're facing an issue with openvswitch2.15:
running this command on 2 different nodes on the same network leads to both ports having the same MAC address, leading to obvious networking issues:

ovs-vsctl add-port br-ex vlan103 tag=103 -- set interface vlan103 type=internal

In order to ensure it's not already patched, we've upgraded openvswitch to this build:
https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1938775

Some more notes:
- env is running on libvirt (multiple VMs on libvirt)
- env is using a single bridge exposed from the hypervisor
- env is running with latest EL9 qcow2 image and latest "overcloud-hardened-uefi-full" image provided by OSP.

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Get 2 nodes on the same network
2. ovs-vsctl add-port br-ex vlan103 tag=103 -- set interface vlan103 type=internal
3. check the MAC addresses on those ports (ip link show vlan103)

Actual results:
the MAC addresses are the same on both nodes, preventing any networking to actually work

Expected results:
MAC addresses should be different

Additional info:
This happens when trying to deploy an OSP-17 overcloud with latest compose available.

Comment 1 Cédric Jeanneret 2022-03-23 14:46:16 UTC

(note: I can't set the "Version" to the right value since "RHEL 9.0" doesn't exist in the list)

Comment 2 Cédric Jeanneret 2022-03-24 13:34:50 UTC

Hello,

Closing this, found out the actual issue:
- I'm editing a base image using virt-sysprep
- that base image is then used to provision the other VMs

According to https://bugzilla.redhat.com/show_bug.cgi?id=1554546, virt-customize (as well as virt-sysprep) generates a machine-id; since the image is then used to provision a bunch of VM, they end up with the same machine-id.

So it appears OVS is using the machine-id under some conditions in order to generate MAC for vlan ports. At least, that's my thinking, since ensuring I get different machine-id on the VMs shows different MAC addresses being generated, leading to a working network now.

Sorry for the noise.

Cheers,

C.