Bug 2067216

Summary: [EL9] add-port creates interfaces with the same MAC address on different nodes
Product: Red Hat Enterprise Linux Fast Datapath Reporter: Cédric Jeanneret <cjeanner>
Component: openvswitch2.15Assignee: Open vSwitch development team <ovs-team>
Status: CLOSED NOTABUG QA Contact: ovs-qe
Severity: urgent Docs Contact:
Priority: urgent    
Version: RHEL 9.0CC: apevec, ctrautma, jhsiao, ralongi
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-03-24 13:34:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Cédric Jeanneret 2022-03-23 14:44:50 UTC
Description of problem:
While testing EL9 and OSP-17, we're facing an issue with openvswitch2.15:
running this command on 2 different nodes on the same network leads to both ports having the same MAC address, leading to obvious networking issues:

ovs-vsctl add-port br-ex vlan103 tag=103 -- set interface vlan103 type=internal

In order to ensure it's not already patched, we've upgraded openvswitch to this build:

Some more notes:
- env is running on libvirt (multiple VMs on libvirt)
- env is using a single bridge exposed from the hypervisor
- env is running with latest EL9 qcow2 image and latest "overcloud-hardened-uefi-full" image provided by OSP.

Version-Release number of selected component (if applicable):

How reproducible:

Steps to Reproduce:
1. Get 2 nodes on the same network
2. ovs-vsctl add-port br-ex vlan103 tag=103 -- set interface vlan103 type=internal
3. check the MAC addresses on those ports (ip link show vlan103)

Actual results:
the MAC addresses are the same on both nodes, preventing any networking to actually work

Expected results:
MAC addresses should be different

Additional info:
This happens when trying to deploy an OSP-17 overcloud with latest compose available.

Comment 1 Cédric Jeanneret 2022-03-23 14:46:16 UTC
(note: I can't set the "Version" to the right value since "RHEL 9.0" doesn't exist in the list)

Comment 2 Cédric Jeanneret 2022-03-24 13:34:50 UTC

Closing this, found out the actual issue:
- I'm editing a base image using virt-sysprep
- that base image is then used to provision the other VMs

According to https://bugzilla.redhat.com/show_bug.cgi?id=1554546, virt-customize (as well as virt-sysprep) generates a machine-id; since the image is then used to provision a bunch of VM, they end up with the same machine-id.

So it appears OVS is using the machine-id under some conditions in order to generate MAC for vlan ports. At least, that's my thinking, since ensuring I get different machine-id on the VMs shows different MAC addresses being generated, leading to a working network now.

Sorry for the noise.