Bug 1424641

Summary: Team MAC address changes after reboot or a down/up cycle
Product: Red Hat Enterprise Linux 7 Reporter: noah davids <ndavids>
Component: NetworkManagerAssignee: Beniamino Galvani <bgalvani>
Status: CLOSED ERRATA QA Contact: Desktop QE <desktop-qa-list>
Severity: low Docs Contact:
Priority: unspecified    
Version: 7.3CC: 674791149, atragler, bgalvani, fgiudici, lrintel, rkhan, sukulkar, thaller, vbenes
Target Milestone: rcKeywords: Reopened
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: NetworkManager-1.8.0-0.4.rc1.el7 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-01 09:24:38 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description noah davids 2017-02-17 22:24:36 UTC
Description of problem:
The Team MAC address seems to randomly change after every reboot or when the team is down/up cycled

Version-Release number of selected component (if applicable):

kernel-3.10.0-514.el7.x86_64
NetworkManager-team-1.4.0-12.el7.x86_64
NetworkManager-1.4.0-12.el7.x86_64


How reproducible:
100 %

Steps to Reproduce:

for x in 1 2 3 4 5; do nmcli conn down team0; sleep 5 ; nmcli conn up team0; ip l l team0; done | grep link/ether
    link/ether d2:97:a4:3e:b5:65 brd ff:ff:ff:ff:ff:ff
    link/ether 56:89:b3:8c:7b:f8 brd ff:ff:ff:ff:ff:ff
    link/ether 1a:e7:a9:c2:c2:2c brd ff:ff:ff:ff:ff:ff
    link/ether 66:10:0a:e6:d5:ed brd ff:ff:ff:ff:ff:ff
    link/ether 52:94:32:6a:98:c5 brd ff:ff:ff:ff:ff:ff

Actual results:
MAC address changes after every cycle

Expected results:
MAC address should remain the name

Additional info:

The MACADDR option is in the ifcfg-team0 config but it doesn't seem to work

# grep ADDR /etc/sysconfig/network-scripts/ifcfg-team0
IPV6_ADDR_GEN_MODE=stable-privacy
IPADDR=172.16.1.207
MACADDR=02:04:08:10:20:40

This seems to be related to Bug 1387506 - Bond interface MAC address is not persistent after reboot. except that according to comment 7 of that bug this should no longer be an issue in the releases I am running (assuming that the same bug fix would fix both bonding and teaming). 

Work around:

Add a file to /etc/NetworkManager/dispatcher.d

# cat /etc/NetworkManager/dispatcher.d/30-team0-link 
if [ $2 == "down" ]
then
   exit
fi

if [ $1 == "team0" ]
then
   /sbin/ip link set address 02:01:02:03:04:05 dev team0
fi


Make sure that the script is owned by root and not writeable by any other user or group, as per the NetworkManager man page in RHEL7.  The following commands can assist with this:

chown root /etc/NetworkManager/dispatcher.d/30-team0-link
chmod u+x /etc/NetworkManager/dispatcher.d/30-team0-link
chmod go-w /etc/NetworkManager/dispatcher.d/30-team0-link

Comment 2 Thomas Haller 2017-02-20 17:21:03 UTC
this is mostly a dupe of bug 1420708.

this is also related to bug 1386872, where in rhel-7.4 you will be able to set the MAC address for virtual devices, like team.

Comment 3 noah davids 2017-02-20 17:43:12 UTC
I would disagree that this is a dup of bug 1420708. 1420708 states that the MAC address should be based on the alphabetic order of the slave names. This would eliminate the randomness of the MAC addresses, assuming any fix is applied to teams as well as bonds BUT would not eliminate the issue of being about to assign a fixed address. It seems more related to 1386872. But a comment in that bug specifically states "For teams, teamd takes care of setting a MAC address for the device and so the desired value should be set in the JSON team configuration." Which does imply that the bug isn't needed BUT I cannot find anything that tells me what the jason config would look like.

Comment 4 noah davids 2017-02-20 23:29:29 UTC
Adding a hwaddr entry to the team config can be used to set the team MAC address.

# cat ifcfg-team0 | grep TEAM_CONFIG
TEAM_CONFIG="{ 	\"device\":		\"team0\", 	\"hwaddr\": \"02:02:02:02:02:02\",    \"runner\": { 		\"name\": \"lacp\", 		\"active\": true, 		\"fast_rate\": true, 		\"tx_hash\": [\"eth\", \"ipv4\", \"ipv6\"] }, 	\"link_watch\":		{\"name\": \"ethtool\"}, 	\"ports\":		{\"enp9s0f0\": {}, \"enp9s0f1\": {}, \"eno1\": {}} }"

Note after making the change you need to tell Network manager to reload the configuration

nmcli conn reload team0

# for x in 1 2 3 4 5; do nmcli conn down team0; sleep 5 ; nmcli conn up team0; ip l l team0; done | grep link/ether
    link/ether 02:02:02:02:02:02 brd ff:ff:ff:ff:ff:ff
    link/ether 02:02:02:02:02:02 brd ff:ff:ff:ff:ff:ff
    link/ether 02:02:02:02:02:02 brd ff:ff:ff:ff:ff:ff
    link/ether 02:02:02:02:02:02 brd ff:ff:ff:ff:ff:ff
    link/ether 02:02:02:02:02:02 brd ff:ff:ff:ff:ff:ff

I am changing the status to closed not a bug since there is a configuration setting that eliminates the problem.

Comment 5 Thomas Haller 2017-02-20 23:40:13 UTC
Maybe NM still should handle this better.
At least, there has a ethernet.cloned-mac-address settings, which apparently doesn't work.

Beniamino, what do you think?

Comment 6 Beniamino Galvani 2017-02-21 09:53:29 UTC
(In reply to Thomas Haller from comment #5)
> Maybe NM still should handle this better.
> At least, there has a ethernet.cloned-mac-address settings, which apparently
> doesn't work.
> 
> Beniamino, what do you think?

It's not that easy as with other devices, because teamd tries to
change the MAC address of the team device at various points, for
example each time a port is added or according to the
runner.hwaddr_policy configuration. So, just setting the MAC of the
team from NM before starting teamd won't work.

A cleaner alternative would be to inject the "hwaddr" property in the
JSON configuration (and throw a warning/error if it is already present
with a different value), so that teamd will manage the MAC by itself:
when "hwaddr" is explicitly set, teamd locks it to the given value
[1]. But this solution depends on the availability of Jansson compiled
in, as we need it to parse and modify the configuration. When Jansson
is not available, maybe we can just ignore the cloned-mac property and
print a warning.

Thomas, what's your opinion?

[1] https://github.com/jpirko/libteam/blob/master/teamd/teamd.c#L868

Comment 7 Thomas Haller 2017-02-21 10:10:39 UTC
(In reply to Beniamino Galvani from comment #6)
> (In reply to Thomas Haller from comment #5)
> > Maybe NM still should handle this better.
> > At least, there has a ethernet.cloned-mac-address settings, which apparently
> > doesn't work.
> > 
> > Beniamino, what do you think?
> 
> It's not that easy as with other devices, because teamd tries to
> change the MAC address of the team device at various points, for
> example each time a port is added or according to the
> runner.hwaddr_policy configuration. So, just setting the MAC of the
> team from NM before starting teamd won't work.
> 
> A cleaner alternative would be to inject the "hwaddr" property in the
> JSON configuration (and throw a warning/error if it is already present
> with a different value), so that teamd will manage the MAC by itself:
> when "hwaddr" is explicitly set, teamd locks it to the given value
> [1]. But this solution depends on the availability of Jansson compiled
> in, as we need it to parse and modify the configuration. When Jansson
> is not available, maybe we can just ignore the cloned-mac property and
> print a warning.
> 
> Thomas, what's your opinion?

Injecting the hwaddr into the json sounds like a good solution.

We could make libjansson a strict requirement of the team plugin, that seems reasonable, as also teamd has that requirement already.

On the other hand, it should still be possible to build libnm without linking against libjansson.

Comment 8 Beniamino Galvani 2017-02-22 21:15:19 UTC
Please review bg/team-cloned-mac-rh1424641.

Comment 9 Thomas Haller 2017-02-22 23:21:39 UTC
(In reply to Beniamino Galvani from comment #8)
> Please review bg/team-cloned-mac-rh1424641.

very nice. Pushed some fixups.

lgtm

Comment 10 Beniamino Galvani 2017-02-23 09:34:16 UTC
Squashed the fixups and merged to master as:

https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=36bb22f598b6b25503041bcbc283aa21510f3754

Thanks!

Comment 12 errata-xmlrpc 2017-08-01 09:24:38 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2017:2299