There is not a lot of information in this bug. I admittedly don't have a switch setup, but I'm not immediately able to reproduce given the provided information.

My dumb method for reproducing losing an interface is `virsh domif-setlink <domain> <MAC> down`.

Can you provide more information?

Hi Mabe,

Given that we use physical interfaces, as you can see, there is a bond configuration created using two physical interfaces


[core@r192a-csah-pri ~]$ ssh r192ac3 ip a s ens2f0
2: ens2f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 3c:fd:fe:d1:b5:60 brd ff:ff:ff:ff:ff:ff

[core@r192a-csah-pri ~]$ ssh r192ac3 ip a s ens3f0
4: ens3f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 3c:fd:fe:d1:b5:60 brd ff:ff:ff:ff:ff:ff

[core@r192a-csah-pri ~]$ ssh r192ac3 ip a s bond0
9: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 3c:fd:fe:d1:b5:60 brd ff:ff:ff:ff:ff:ff
    inet 192.168.192.78/26 brd 192.168.192.127 scope global noprefixroute bond0
       valid_lft forever preferred_lft forever

Note: ens2f0 and ens3f0 are used to create bond0.

Below is what I have used to create bonding with active-backup mode

menuentry 'r192ac3' --class fedora --class gnu-linux --class gnu --class os {
  linuxefi rhcos/4.6/rhcos-live-kernel-x86_64 nomodeset rd.neednet=1 coreos.inst.insecure coreos.live.rootfs_url=http://<http:port>/rhcos/4.6/rhcos-live-rootfs.x86_64.img coreos.inst=yes coreos.inst.install_dev=nvme0n1 coreos.inst.image_url=http://<http:port>/rhcos/4.6/rhcos-metal.x86_64.raw.gz coreos.inst.ignition_url=http://<http:port>/ignition/worker.ign ip=<node ip>::<gateway>:<subnet>:<hostname>:bond0:none bond=bond0:ens2f0,ens3f0:mode=active-backup nameserver=<dns>
  initrdefi rhcos/4.6/rhcos-live-initramfs.x86_64.img
}


Once the node is up and running, I have started a ping

PING r192ac3.oss.labs (192.168.192.78) 56(84) bytes of data.
64 bytes from r192ac3.oss.labs (191.168.192.78): icmp_seq=1 ttl=64 time=0.227 ms
64 bytes from r192ac3.oss.labs (192.168.192.78): icmp_seq=2 ttl=64 time=0.204 ms
64 bytes from r192ac3.oss.labs (192.168.192.78): icmp_seq=3 ttl=64 time=0.169 ms


Now, while the ping is going on, executed a shutdown on the primary interface (ens2f0) from the switch side as below.

S5232-1(conf-if-eth1/1/4:3)# show configuration
!
interface ethernet1/1/4:3
 no shutdown
 channel-group 15 mode active
 no switchport
 mtu 9216
 flowcontrol receive on
 flowcontrol transmit on
 lacp port-priority 32015
S5232-1(conf-if-eth1/1/4:3)# shutdown

Below is the output from RHCOS side to ensure that right interface is shutdown. Please notice that ens2f0 is in DOWN state.

[core@r192a-csah-pri uefi]$ ssh r192ac3 ip a s ens2f0
2: ens2f0: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc mq master bond0 state DOWN group default qlen 1000
    link/ether 3c:fd:fe:d1:b5:60 brd ff:ff:ff:ff:ff:ff
[core@r192a-csah-pri uefi]$ ssh r192ac3 ip a s ens3f0
4: ens3f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master bond0 state UP group default qlen 1000
    link/ether 3c:fd:fe:d1:b5:60 brd ff:ff:ff:ff:ff:ff
[core@r192a-csah-pri uefi]$ ssh r192ac3 ip a s bond0
8: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 3c:fd:fe:d1:b5:60 brd ff:ff:ff:ff:ff:ff
    inet 192.168.192.78/26 brd 100.67.192.127 scope global noprefixroute bond0
       valid_lft forever preferred_lft forever


At this point, upon executing "no shutdown" in the switch, things fail and loose connectivity to the node (compute node in this case). Refer to primary_interface_no-shutdown.png file to see that RHCOS does recognize the interface being up, but for some odd reason, we loose connectivity and pings fail

Only way to bring the system connectivity back again is, by shutting down the primary interface in the switch again

Let me know if you want to schedule a session and I can show the error

Created attachment 1784224 [details]
primary_interface_no-shutdown

Summary:

- system is up (bond is up with both devices)
- take down ens2f0 interface on the switch side
    - secondary interface (ens3f0) is promoted, things still work
- bring up ens2f0 interface on the switch side
    - system loses connectivity


I'm doing exactly that setup (without the switch) and it seems to work fine. This indicates to me
that the test I'm doing is not sufficiently equivalent, so a screen share might be useful.

I'm willing to do a screen share, but there are some other data points that would be useful.

- Can you reproduce this with RHEL?

This would at least let us know what teams need to be engaged.

(In reply to Dusty Mabe from comment #4)
> Summary:
> 
> - system is up (bond is up with both devices)
> - take down ens2f0 interface on the switch side
>     - secondary interface (ens3f0) is promoted, things still work
> - bring up ens2f0 interface on the switch side
>     - system loses connectivity
> 
> 
> I'm doing exactly that setup (without the switch) and it seems to work fine.
> This indicates to me
> that the test I'm doing is not sufficiently equivalent, so a screen share
> might be useful.

Perfect. Let me know when you have some time and we can work. lets take this through email conversations.
> 
> I'm willing to do a screen share, but there are some other data points that
> would be useful.
> 
> - Can you reproduce this with RHEL?
I dont have cycles to test this now. We can give this a try during our screenshare.
> 
> This would at least let us know what teams need to be engaged.

Hi Mabe, I see the same pattern in RHEL 8.2 as well.

Only when the secondary interface is shutdown on switch side, things are working with primary interface taking control. I could not copy the config files from RHCOS as is because /etc/NetworkManager/system-connections directory was empty to begin with. So used the nmcli commands to create bonding with mode "active-backup".

I also tried with mode active-backup and miimon=100 but it made no difference. FYI, I checked different cluster using different switches with same problem as well.

Let me know if you need any further information.

If you could share the following it would help when we try to engage others:

- the exact versions of some software on your RHEL 8.2 machine. kernel, NetworkManager, systemd, etc
- the nmcli commands you ran on the RHEL 8.2 machine
- a description of your Network setup including the switch setup for the two separate networks the interfaces are attached to


It would also really help if we knew if this issue was resolved in a future release. Trying with latest RHEL 8.3 content could help find the answer to that if you have the ability.

Hi Mabe, below is the information you asked for.

[root@rc3 ~]# yum list NetworkManager
Installed Packages
NetworkManager.x86_64                                                                1:1.22.8-4.el8                                                                 @anaconda

[root@rc3 ~]# yum list systemd
Installed Packages
systemd.x86_64                                                                      239-29.el8                                                                      @anaconda

[root@r192ac3 ~]# yum list kernel
Installed Packages
kernel.x86_64                                                                    4.18.0-193.el8                                                                     @anaconda

Commands used to configure bonding are below

    5  nmcli connection add type bond ifname bond0 con-name bond0 bond.options "mode=active-backup"
    6  nmcli connection add type ethernet ifname ens2f0 master bond0
    7  nmcli connection add type ethernet ifname ens3f0 master bond0
    8  nmcli connection modify bond0 ipv4.method manual ipv4.addresses <ip/network cidr> ipv4.gateway <gw> ipv4.dns <dns ip> ipv4.dns-search <domain> connection.autoconnect yes
    9  nmcli connection up bond0


regarding switch setup

- switch 1 is connected to port 1 in NIC 1
- switch 2 is connected to port 1 in NIC 2
- Port channel configured for these two interface at switch side (in both switches)

Please refer to the first comment from me regarding the switch configs

Hey Umesh,

I'm engaging the NetworkManager team to see if we can get to the bottom of what is happening. 

NM team,

I tried to reproduce this bug locally in a virtual machine with a dumb (inappropriate) physical setup for bonding. When I took the link down and brought it back up things went fine. IOW I could not reproduce.

I did have a screen share with Umesh last week and can confirm the behavior he describes does seem to be happening. Basically initial failover when he takes the primary link down works fine, but when he brings that link back up the system loses all connectivity. Can't SSH in. No DNS. Can't ping gateway. NM seems to think things are up and running fine.


Umesh,

More data is always helpful. If you have time this information would be useful:

- dmesg output and journal logs from NetworkManager (journalctl -u NetworkManager -b0) ?
- can you reproduce this on RHEL 8.3 ?

Hi Mabe, got tied up another work. I believe there is 8.3 available now and as well as 8.4. Let me know if you still need me to try the active-backup setup with latest RHEL OS.

Two workarounds I can see working are below

Workaround 1:

- after the primary interface is down and then comes up, issue arises. At this time, shutdown the backup interface to ensure primary interface becomes active interface again.
- Once the primary interface becomes active again, it is OK to bring the backup interface up again

Workaround 2:

- after the primary interface is down and then comes up, issue arises. At this time, rebooting (cold) the node works.

I would be interesting to see the details of the system at the momentwhen the connectivity is broken.
Would you be able to show:

  ip -d link
  ip addr
  grep -R ^ /sys/devices/virtual/net/*/bonding/

I'm not familiar with Cisco switches configuration, but doesn't "channel-group 15 mode active" enable LACP? If so, wouldn't "mode=802.3ad" be more appropriate in the kernel command line, instead of "active-backup"?

does comment 14 help?

Regarding comment #14, yes, it is recommended to have like mode=802.3ad. But there is another issue that is currently preventing us from using that. https://bugzilla.redhat.com/show_bug.cgi?id=1960750

once the change is pushed, which I was told should be mid July or so as per the BZ https://bugzilla.redhat.com/show_bug.cgi?id=1960750, then we will move to mode=802.3ad and at that point of time, this issue is no longer a problem.

(In reply to umesh_sunnapu from comment #16)
> Regarding comment #14, yes, it is recommended to have like mode=802.3ad. But
> there is another issue that is currently preventing us from using that.
> https://bugzilla.redhat.com/show_bug.cgi?id=1960750
> 
> once the change is pushed, which I was told should be mid July or so as per
> the BZ https://bugzilla.redhat.com/show_bug.cgi?id=1960750, then we will
> move to mode=802.3ad and at that point of time, this issue is no longer a
> problem.

Thank you for the update. From what I understand, there is nothing to be done in NetworkManager, therefore I close this bug. Please re-open this bug or file a new one if necessary.

*** This bug has been marked as a duplicate of bug 1960750 ***