Bug 2216056

Hi,

the issue seems to be that NM starts sending the ARP announcements when the team interface still has a random MAC. Shortly after, a port gets attached to the team, and NM sends the remaining ARPs with the wrong (random) MAC.

What should happen instead is that NM waits the team to have a port attached and that the team's MAC address is inherited from one port; then it can send ARP announcements.

This problem already appeared in the past, with the following timeline:

 - it was initially reported as [1] in RHEL 7.6 and fixed in RHEL 7.7 (NetworkManager 1.16);

 - then we found that the fix was incomplete [2] and a new fix was developed in RHEL 8.5 (NetworkManager 1.32.10);

 - later, in RHEL 8.6 (NetworkManager 1.36) there was a rework of the handling of IP configuration in NetworkManager.

In the case linked to this bz I see that the issue is reproducible with 1.34 and not with 1.36; this suggests that the IP configuration rework might have fixed the problem. Unfortunately, that rework can't be backported to RHEL7 because it is too big. On the other hand, the fix present in [2] should be easy to backport; therefore it would be useful to understand why it is not working as expected.

Would it be possible to have a 'journalctl -b' log with NetworkManager set at TRACE level, for both the following NM versions?

 - 1.32 or 1.34 (where the issue is reproduced)
 - 1.36 (where the issue is NOT reproduced)

Do you have a setup to reproduce the issue? If so, can you share the script or the environment?

Thank you.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1678796
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1956793

Hello @bgalvani ,

Thanks for checking this.

The issue is easily reproducible with below steps:

1. Start the below script on the system:

#!/bin/bash
TEAM=team1
PORT=ens7s0
IP="192.168.11.11/24"
let i=1
while [ 1 ] ; do
	date
	echo "Test # $i"
	nmcli con down $TEAM; nmcli con down $PORT; nmcli con del $TEAM; nmcli con del $PORT;
        sleep 1;
  	nmcli con add type team ifname $TEAM con-name $TEAM team.runner activebackup ipv4.method manual ipv4.addresses $IP ipv6.method ignore
        sleep 1
	ip a show
        nmcli con add type team-slave ifname $PORT con-name $PORT master $TEAM
	sleep 3
        let i+=1
done


2. Watch $PORT externally (e.g.) (we used host bridge virbr0)
   tshark -i virbr0 -Y "arp.src.hw_mac != eth.src"


Are you okay with the above reproduction steps? If you still want us to generate TRACE logs from our VM, do let us know.


Thanks,
Prijesh

Thanks, I can reproduce the bug with your script. The fix in 1.34 [1] works well with bonding but has issues with teaming. When a port is attached to the team interface, carrier goes up on the team while the interface still has the random MAC. After some time, the userspace teamd process selects the active port and then the MAC gets updated. Example log:

          ### no carrier, random MAC
  <trace> [1687552518.0684] platform-linux: event-notification: RTM_NEWLINK, flags 0, seq 0: 23: team1 <UP;broadcast,multicast,up> mtu 1500 arp 1 team* not-init addrgenmode none addr 2E:8A:C5:4A:AC:F2 brd FF:FF:FF:FF:FF:FF rx:0,0 tx:0,0
  <debug> [1687552519.1281] platform: (enp7s0) link: enslaving to master 'team1'
          ### carrier goes up
  <debug> [1687552519.1283] platform: (team1) signal: link changed: 23: team1 <UP,LOWER_UP;broadcast,multicast,up,lowerup> mtu 1500 arp 1 team* init addrgenmode eui64 addr 2E:8A:C5:4A:AC:F2 brd FF:FF:FF:FF:FF:FF driver team rx:0,0 tx:0,0
  <info>  [1687552519.1289] device (team1): carrier: link connected
          ### start sending ARP with wrong MAC
  <debug> [1687552519.1291] acd[0x7fed74007ca0,23]: announcing address 192.168.11.11 (hw-addr 2E:8A:C5:4A:AC:F2)
  teamd_team1[9427]: Found best port: "enp7s0" (ifindex "3", prio "0").
          ### now the MAC is updated
  <trace> [1687552519.1311] platform-linux: event-notification: RTM_NEWLINK, flags 0, seq 0: 23: team1 <UP,LOWER_UP;broadcast,multicast,up,lowerup> mtu 1500 arp 1 team* not-init addrgenmode eui64 addr 52:54:00:2D:F5:1D brd FF:FF:FF:FF:FF:FF rx:0,0 tx:0,0
  
1.36 fixes that problem by restarting ARP announcements when the MAC of the team changes. However, the fix in 1.36 can't be backported because it is part of a big rework. A workaround to the problem would be to set a static MAC on the team interface, as in:

  nmcli connection modify $TEAM ethernet.cloned-mac-address $(cat /sys/class/net/$PORT/address)

In this way, the team interface is created from the beginning with the right MAC. Would it be an acceptable solution?

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1956793

Hello @bgalvani 

Thanks for digging this further.

I will be sharing the information with the customer and will inform here as soon as we get feedback from them.

I have tried these steps in my VM and it looks promising. 

Just to check with you, I have given them the below workaround as set "team.notify-peers-count 10 team.notify-peers-interval 1000", do you think that is a good option:

+++
I have tested in my local system with the above, and observed(tested with 70+ attempts): 

1st g-ARP sent with correct MAC
2nd g-ARP was sent with random MAC
remaining g-ARP was sent with the correct MAC
 
As the last many packets are sent with the correct MAC, it will override the previous random MAC and can help

+++

Just wanted to ask the above, so that we can add a possible workaround in KCS that we have created for this issue.


Thanks,
Prijesh

Yes, it seems notify-peers could help to make sure the right MAC is announced after the random one. Another alternative would be to delay the ARP announcements by enabling DAD (duplicate address detection) for IPv4. To do that, set for example `ipv4.dad-timeout 3000`. As said, another approach would be to set a fixed MAC on the team.

Till Maas,

I have asked the customer about the workarounds but have not got any response yet.

Once I get a response, I will update here.


Thanks,
Prijesh

Hello,

We got a response as their end customer is not back after our workarounds were provided, so I feel there is no need to keep this BZ open, we can close it.

Thanks,
Prijesh

Thanks, since the issue is already fixed in RHEL 8.6 and later, I'm closing the bz.