Could you enable debug logging of NetworkManager (level=TRACE) and provide a logfile? Configure the loglevel in /etc/NetworkManager/NetworkManager.conf and reproduce the issue.


I think the problem is that the permanent MAC address is read via ethtool, for which we specify the interface by name, not by ifindex.

That might be somewhat avoided, by ensuring we don't read the permanent MAC address before UDEV has done it's business. Of course, the interface could still be renamed later on, and the race still exists.

Created attachment 1213900 [details]
log from 2nd failure

Thomas - thanks for looking at this.  I will get logs with the updated log level but it may take a few days.  This problem has only been seen in Dell's configuration and, as its a ramdisk image, we don't have the ability to change the NetworkManager.conf 
without rebuilding a new image for them and having them redeploy.  In addition the Dell contact (Chris Dearborn) is out this week.

For reference, this is the Openstack bug from Dell that describes the original issue - https://bugzilla.redhat.com/show_bug.cgi?id=1384187

As described in that bug, this problem is intermittent and appears to be timing related.  It worked fine on 4 of the 6 nodes, on the other node that failed it was the mac of p6p2 instead of p4p1 that was used for em1.  I've included the journal of this 2nd case in case its useful to see the differences.

I would also be interested to know of of any suggested workarounds, for example I assume that disabling interface renaming would resolve this, e.g. via kernel flags 'net.ifnames=0' and 'biosdevname=0.  That however could lead to different Openstack problems as the interface names between deployed images may be different.

Hi,

if it's too cumbersome, don't bother with the logfiles. I think the issue is clear. Thanks.



The problem is that NetworkManager caches the permanent MAC address and only attempts to read it once.


NM gets the permanent MAC address via ethtool ioctl, which has the interface name as argument. Thus, there is a race of using the wrong name. The real fix for this would be to use a different kernel API that uses the ifindex (which currently does not exist).

The race is especially likely before UDEV is finished renaming the device. Later on, the device usually doesn't get renamed and is much more unlikely that NetworkManager hits the race.


A part of the problem is, that NM already tries to read the permanent MAC address before UDEV is done renaming the interface.
Another reason why that is bad (not related to this bug) is for software devices without permanent MAC address (veth), NM takes the current MAC address and caches it (using it as pseudo permanent address). The user would be advised to configure the initial MAC address of the device via UDEV, so NM should not read the MAC address before UDEV completed.



Due to the race you end up with NetworkManager thinking that a certain device has the wrong permanent MAC address. Workarounds are:

(1) don't rename the interfaces :)

(2) avoid NetworkManager using the wrong MAC address:

(2a) NetworkManager uses it for example for configurations in /etc/NetworkManager/NetworkManager.conf (see Device List Format in `man NetworkManager.conf`):

  [keyfile]
  unmanaged-devices=mac:A0:36:9F:14:49:90

use instead the interface name there.

(2b) There is the per-connection setting ethernet.cloned-mac-address (see `nmcli connection show $NAME`). Don't set it to "permanent", because then NetworkManager will try to configure the (wrong) permanent address on the device. Set it either to an explicit MAC address or "preserve" to not change it.

  nmcli connection modify $NAME ethernet.cloned-mac-address preserve

Note, that the property has by default the value "", which means to fallback to a globally configured default value. you can configure that in a file like /etc/NetworkManager/conf.d/90-cloned-mac-address-preserve.conf

  [connection-cloned-mac-address-preserve]
  ethernet.cloned-mac-address=preserve

Followed by `killall -SIGHUP NetworkManager`.
See "Connection section" in `man NetworkManager.conf` for details.
This will then take effect for all connections that have their per-connection value unset.






There is upstream branch th/preserve-fake-perm-hwaddr-bgo772880 on review to delay reading the permanent MAC address before UDEV is done.

fix merged to upstream master: https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=e2b7da7b8202a3f9da1b73d78f5d210afa610a3e

also backported to upstream nm-1-4 branch:
https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=221851436b3bc5c3b43f574269adee9ca052a50d

Would it be possible to get a 7.3-z patch for this?

We need qa and pm acks before we can propose for z-stream.

(In reply to Karl Hastings from comment #9)
> We need qa and pm acks before we can propose for z-stream.

Rerouting to Vlad.

Bob, any chance to test scratch-build https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=12071007 ?

Thomas - we've tested the patch on two different systems that were exhibiting the issue earlier and have verified that it works with this patch.

These are the rpms used:
- NetworkManager-1.4.0-13.test.rh1388286.01.el7_3.x86_64.rpm
- NetworkManager-libnm-1.4.0-13.test.rh1388286.01.el7_3.x86_64.rpm
- NetworkManager-tui-1.4.0-13.test.rh1388286.01.el7_3.x86_64.rpm
- NetworkManager-config-server-1.4.0-13.test.rh1388286.01.el7_3.x86_64.rpm
- NetworkManager-team-1.4.0-13.test.rh1388286.01.el7_3.x86_64.rpm

*** Bug 1384187 has been marked as a duplicate of this bug. ***

*** Bug 1400764 has been marked as a duplicate of this bug. ***

I am seeing this in latest OSP10. Do we know if we have a fix yet?

(In reply to Sai Sindhur Malleni from comment #28)
> I am seeing this in latest OSP10. Do we know if we have a fix yet?

Update should be out in 7.3.3 z stream. Not sure how this affects OSP10. This bug tracks 7.4.

I tested with RHEL 7.4 and the following NetworkManager rpms:
NetworkManager-tui-1.8.0-9.el7.x86_64
NetworkManager-config-server-1.8.0-9.el7.noarch
NetworkManager-team-1.8.0-9.el7.x86_64
NetworkManager-libnm-1.8.0-9.el7.x86_64
NetworkManager-1.8.0-9.el7.x86_64

With the agent.ramdisk using these rpms, I ran introspection multiple times. I did not see any problems with interface renaming causing incorrect mac addresses to be set.

Marking as verified according to previous comment.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2299