RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1740557 - MAC address issue with an OvS bridge
Summary: MAC address issue with an OvS bridge
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 7
Classification: Red Hat
Component: NetworkManager
Version: 7.7
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: rc
: 7.6
Assignee: sushil kulkarni
QA Contact: Desktop QE
URL:
Whiteboard:
Depends On:
Blocks: 1791672 1791674
TreeView+ depends on / blocked
 
Reported: 2019-08-13 09:11 UTC by Denis Ollier
Modified: 2020-03-31 20:08 UTC (History)
17 users (show)

Fixed In Version: NetworkManager-1.18.4-3.el7
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1791672 1791674 (view as bug list)
Environment:
Last Closed: 2020-03-31 20:07:59 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2020:1162 0 None None None 2020-03-31 20:08:37 UTC

Description Denis Ollier 2019-08-13 09:11:32 UTC
Description of problem
----------------------

Following https://bugzilla.redhat.com/show_bug.cgi?id=1733709, I create an OvS bridge on BM servers running RHEL 7.7 using nmcli.

First NIC (default_device) and second NIC (secondary_device) of the server are attached to a bond which is attached to the bridge.

The bridge is configured to retrieve its IP address from DHCP. To make sure the bridge received the IP address is the one matching the server DNS record, the bridge is configured to use the MAC address of the first NIC (default_device) of the server:

>    mac=$(nmcli -g GENERAL.HWADDR dev show $default_device | sed -e 's/\\//g')
>    
>    # make bridge
>    nmcli conn add type ovs-bridge conn.interface brcnv
>    nmcli conn add type ovs-port conn.interface port0 master brcnv
>    nmcli conn add type ovs-port conn.interface brcnv-port master brcnv
>    nmcli conn add type ovs-interface conn.id brcnv-iface conn.interface brcnv master brcnv-port ipv4.method auto 802-3-ethernet.cloned-mac-address $mac connection.autoconnect no
>    
>    # make bond
>    nmcli conn add type ovs-port conn.interface bond0 master brcnv ovs-port.bond-mode balance-slb
>    nmcli conn add type ethernet conn.interface $default_device master bond0
>    nmcli conn add type ethernet conn.interface $secondary_device master bond0
>    nmcli conn down "System $default_device"
>    nmcli conn mod "System $default_device" connection.autoconnect no
>    nmcli conn up brcnv-iface
>    nmcli conn mod brcnv-iface connection.autoconnect yes

However after rebooting the server, the bridge interface sometimes receive an IP address not linked to the MAC address of the first NIC.

tcpdump captures seem to show that the MAC address used by the bridge is "random" while the output of ip addr shows a correct MAC address set to the bridge.

Version-Release number of selected component
--------------------------------------------

NetworkManager and Open vSwitch are installed from RPMs with hotfixes for RHBZ#1733709:

    NetworkManager-1.18.0-5.el7_7.1.x86_64
    NetworkManager-ovs-1.18.0-5.el7_7.1.x86_64
    openvswitch-2.9.0-110.el7fdp.x86_64

How reproducible
----------------

The issue is quite random, which could indicate a race condition issue.

For example, I monitored 3 reboots:

- 1st reboot => wrong lease with MAC address 4e:b7:2e:a8:9d:4d
- 2nd reboot => good lease
- 3rd reboot => wrong lease with MAC address 36:ff:da:be:e0:4b

The MAC address used by the bridge was not the same on those reboots.

Comment 2 Thomas Haller 2019-08-13 09:39:39 UTC
could you provide level=TRACE logs of NetworkManager (or just the entire boot process, no need to pre-filter the available information).

See https://cgit.freedesktop.org/NetworkManager/NetworkManager/tree/contrib/fedora/rpm/NetworkManager.conf#n28 for hints about logging.
See also the comment about journald's rate-limiting, and please disable that first before gathering logs.

Thank you!!

Comment 3 Denis Ollier 2019-08-19 20:07:41 UTC
I tried to gather more logs it last week but I couldn't reproduce this bug on my env anymore.

Closing this ticket for now.

I will reopen it with more logs if the issue comes back.

Comment 4 Denis Ollier 2019-10-24 09:53:24 UTC
Hi,

I could reproduce the issue again today.

Please find attached the systemd journal with NetworkManager full logs.

Version of components used:

    NetworkManager-1.18.0-5.el7_7.1.x86_64
    NetworkManager-ovs-1.18.0-5.el7_7.1.x86_64
    openvswitch-2.9.0-117.el7fdp.x86_64

Thanks.

Comment 6 Beniamino Galvani 2019-10-24 11:53:08 UTC
See also bug 1763734.

Comment 8 Michael Hanulec 2019-10-24 20:02:05 UTC
Appears broken in our openshift 3.11 + rhel7v6 environment too
Please fix

Comment 9 Beniamino Galvani 2019-10-25 09:16:04 UTC
>    # make bridge
>    nmcli conn add type ovs-bridge conn.interface brcnv
>    nmcli conn add type ovs-port conn.interface port0 master brcnv
>    nmcli conn add type ovs-port conn.interface brcnv-port master brcnv
>    nmcli conn add type ovs-interface conn.id brcnv-iface conn.interface brcnv master brcnv-port ipv4.method auto 802-3-ethernet.cloned-mac-address $mac connection.autoconnect no

Changing the MAC address is not implemented at the moment. This is the upstream merge request to add it:

https://gitlab.freedesktop.org/NetworkManager/NetworkManager/merge_requests/321

Note that since the OVS interface has the same name of the OVS bridge, it is considered by ovs as the local interface and so it inherits the MAC from the bridge. Thus, if the MR is merged as is, it will be necessary to set the cloned-mac-address on the ovs-bridge connection, not on the ovs-interface.

> - 1st reboot => wrong lease with MAC address 4e:b7:2e:a8:9d:4d
> - 2nd reboot => good lease
> - 3rd reboot => wrong lease with MAC address 36:ff:da:be:e0:4b

Are you saying that sometimes the MAC address of the ovs-interface is the expected one? If so, do you have logs for that boot?

Comment 10 Denis Ollier 2019-10-25 10:26:03 UTC
(In reply to Beniamino Galvani from comment #9)
> Are you saying that sometimes the MAC address of the ovs-interface is the
> expected one? If so, do you have logs for that boot?

It seems so:

- Wrong lease:

NetworkManager[2219]: <debug> [1571910231.7349] device[0x55bff90390d0] (brcnv): hw-addr: hardware address now BE:DB:42:18:2C:47
NetworkManager[2219]: <debug> [1571910231.7349] device[0x55bff90390d0] (brcnv): hw-addr: update initial MAC address BE:DB:42:18:2C:47
NetworkManager[2219]: <debug> [1571910231.7462] device[0x55bff90390d0] (brcnv): hw-addr: unable to read permanent MAC address (use current: BE:DB:42:18:2C:47)
NetworkManager[2219]: <debug> [1571910231.7973] device[0x55bff90390d0] (brcnv): hw-addr: hardware address now 00:10:9B:35:88:B0

- Good lease:

NetworkManager[2528]: <debug> [1571998564.2603] device[0x555c732717b0] (brcnv): hw-addr: hardware address now 00:10:9B:35:88:B0
NetworkManager[2528]: <debug> [1571998564.2603] device[0x555c732717b0] (brcnv): hw-addr: update initial MAC address 00:10:9B:35:88:B0
NetworkManager[2528]: <debug> [1571998564.2663] device[0x555c732717b0] (brcnv): hw-addr: unable to read permanent MAC address (use current: 00:10:9B:35:88:B0)

See full logs in attached file for wrong and good leases.

Comment 22 David Critch 2019-11-26 17:44:19 UTC
Great, thanks Ben!

Denis, the repo has been updated.

Comment 30 Beniamino Galvani 2019-12-02 16:44:12 UTC
Hi,

NM does not have any special handling of cloned macs, it just sets the
cloned-mac from the ovs-bridge connection into the Bridge table of
ovs, and the cloned-mac from the ovs-interface connection into the
Interface ovs table.

Since the ovs-bridge and the ovs-interface have the same name, ovs
treats the ovs-interface as the 'local interface' and as such, it
inherits the MAC from the bridge. See the 'ovs-vswitchd.conf.db' man
page for more details on how mac addresses are assigned by ovs.

In other words, you should set the cloned mac address on the bridge itself:

 nmcli conn add type ovs-bridge conn.interface brcnv 802-3-ethernet.cloned-mac-address $mac

and it will be inherited by the ovs-interface.

Comment 36 Beniamino Galvani 2020-01-10 15:35:58 UTC
Asking blocker flag for reasons stated in comment 34.

Comment 44 Vladimir Benes 2020-01-15 14:00:18 UTC
two tests added:
1. bridge and iface named the same and cloned mac assigned to bridge
2. bridge and iface named differently and cloned mac assigned to iface

Comment 45 Beniamino Galvani 2020-01-15 14:08:52 UTC
Restoring needinfo? deleted by mistake in comment 44.

Comment 46 sushil kulkarni 2020-01-15 14:31:08 UTC
Hi Anita, 

Can you please provide z stream+ for this?

Thanks!
Sushil

Comment 51 Beniamino Galvani 2020-01-16 12:32:19 UTC
Hi Alona,

this is the official 7.6.z build that includes the fix:

https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=1059407

Comment 53 errata-xmlrpc 2020-03-31 20:07:59 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:1162


Note You need to log in before you can comment on or make changes to this bug.