RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2065049 - NetworkManager can't reapply changes to 'connection.autoconnect-slaves' setting causing loss of connectivity
Summary: NetworkManager can't reapply changes to 'connection.autoconnect-slaves' setti...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: NetworkManager
Version: 8.5
Hardware: x86_64
OS: Linux
unspecified
medium
Target Milestone: rc
: ---
Assignee: Thomas Haller
QA Contact: Matej Berezny
URL:
Whiteboard:
Depends On:
Blocks: 2090344
TreeView+ depends on / blocked
 
Reported: 2022-03-17 08:25 UTC by Juan Orti
Modified: 2022-11-08 11:15 UTC (History)
9 users (show)

Fixed In Version: NetworkManager-1.37.3-1.el8
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-11-08 10:07:32 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-115808 0 None None None 2022-03-17 08:31:00 UTC
Red Hat Knowledge Base (Solution) 6813161 0 None None None 2022-03-17 08:25:40 UTC
Red Hat Product Errata RHBA-2022:7680 0 None None None 2022-11-08 10:08:14 UTC
freedesktop.org Gitlab NetworkManager NetworkManager-ci merge_requests 1025 0 None opened bond: added check if autoconnect-slaves were applied successfully to @bond_reapply 2022-04-20 22:07:56 UTC
freedesktop.org Gitlab NetworkManager NetworkManager merge_requests 1150 0 None opened [th/reapply-autoconnect-slaves] device: allow reapply for changed "connection.autoconnect-slaves" property 2022-03-17 08:55:40 UTC

Description Juan Orti 2022-03-17 08:25:41 UTC 
Description of problem:
If a bond connection changes the value of connection.autoconnect-slaves from -1 (undefined) to 1 (yes), NetworkManager cannot reapply the change to the device and brings the bond interface down an up, causing a loss of connectivity.

This has been observed in a bond of a RHVH hypervisor originally installed with version 4.4.5 (RHEL 8.3):

1. The bond was created at installation time by Anaconda with autoconnect-slaves=-1
2. Further modifications of the connection by RHV using nmstate-0.3.4-27.el8_3.noarch didn't modify the bond, which remained with autoconnect-slaves=-1
3. After upgrading the host to the latest version 4.4.10, nmstate was updated to nmstate-1.1.0-5.el8_5.noarch. In this version, adding a new VLAN to to the bond, caused it to be reconfigured with autoconnect-slaves=1
4. When nmstate applies the new settings, NetworkManager brings down the bond, causing a network outage in the host and in all the VMs running in VLANs dependant of the bond device.


Version-Release number of selected component (if applicable):
NetworkManager-1.32.10-4.el8.x86_64
nmstate-1.1.0-5.el8_5.noarch
kernel-4.18.0-348.12.2.el8_5.x86_64

How reproducible:
Always 

Steps to Reproduce:
1. In a RHEL 8.5 machine, create a bond with connection.autoconnect-slaves unset:

# nmcli con show bond0 |grep connection.autoconnect-slaves
connection.autoconnect-slaves:          -1 (default)

2. Set the option connection.autoconnect-slaves=yes

# nmcli con modify bond0 connection.autoconnect-slaves yes

3. Reapply the settings to the device:

# nmcli device reapply bond0
Error: Reapplying connection to device 'bond0' (/org/freedesktop/NetworkManager/Devices/7) failed: Can't reapply changes to 'connection.autoconnect-slaves' setting

Actual results:
The bond interface and all dependendant interfaces are brought down:

Mar 16 09:13:12 rhvh445-01 NetworkManager[1490]: <info>  [1647421992.9027] audit: op="connection-update" uuid="763cb068-b9bb-4ae5-b483-a5b5befb91de" name="bond0" args="connection.master,connection.autoconnect-slaves" pid=2022 uid=0 result="success"
Mar 16 09:13:12 rhvh445-01 NetworkManager[1490]: <info>  [1647421992.9039] audit: op="connection-update" uuid="206a3829-9919-4958-b9c4-d77e5d46c2cd" name="ovirtmgmt" args="ipv4.route-table,ipv6.route-table" pid=2022 uid=0 result="success"
Mar 16 09:13:12 rhvh445-01 NetworkManager[1490]: <info>  [1647421992.9080] audit: op="device-reapply" interface="ovirtmgmt" ifindex=8 args="ipv4.route-table,ipv6.route-table" pid=2022 uid=0 result="success"
Mar 16 09:13:12 rhvh445-01 NetworkManager[1490]: <info>  [1647421992.9110] audit: op="device-reapply" interface="bond0" ifindex=10 args="connection.master,connection.autoconnect-slaves" pid=2022 uid=0 result="fail" reason="Can't reapply changes to 'connection.master' setting"
Mar 16 09:13:12 rhvh445-01 NetworkManager[1490]: <info>  [1647421992.9124] device (bond0): state change: activated -> deactivating (reason 'new-activation', sys-iface-state: 'managed')
Mar 16 09:13:12 rhvh445-01 kernel: ovirtmgmt: port 1(bond0) entered disabled state
Mar 16 09:13:12 rhvh445-01 NetworkManager[1490]: <info>  [1647421992.9212] device (ovirtmgmt): detached bridge port bond0
Mar 16 09:13:12 rhvh445-01 NetworkManager[1490]: <info>  [1647421992.9212] device (bond0): released from master device ovirtmgmt
Mar 16 09:13:12 rhvh445-01 NetworkManager[1490]: <info>  [1647421992.9218] device (bond0): disconnecting for new activation request.
Mar 16 09:13:12 rhvh445-01 NetworkManager[1490]: <info>  [1647421992.9218] audit: op="connection-activate" uuid="763cb068-b9bb-4ae5-b483-a5b5befb91de" name="bond0" pid=2022 uid=0 result="success"
Mar 16 09:13:12 rhvh445-01 NetworkManager[1490]: <info>  [1647421992.9219] device (enp1s0): state change: activated -> deactivating (reason 'new-activation', sys-iface-state: 'managed')
Mar 16 09:13:12 rhvh445-01 kernel: bond0: (slave enp1s0): Releasing backup interface
Mar 16 09:13:12 rhvh445-01 kernel: bond0: (slave enp1s0): the permanent HWaddr of slave - 52:54:00:18:95:ce - is still in use by bond - set the HWaddr of slave to a different address to avoid conflicts
Mar 16 09:13:12 rhvh445-01 kernel: device enp1s0 left promiscuous mode
Mar 16 09:13:12 rhvh445-01 kernel: bond0: (slave enp8s0): making interface the new active one
Mar 16 09:13:12 rhvh445-01 kernel: device enp8s0 entered promiscuous mode
Mar 16 09:13:12 rhvh445-01 NetworkManager[1490]: <info>  [1647421992.9382] device (bond0): released bond slave enp1s0
Mar 16 09:13:12 rhvh445-01 kernel: 8021q: adding VLAN 0 to HW filter on device enp1s0
Mar 16 09:13:12 rhvh445-01 NetworkManager[1490]: <info>  [1647421992.9416] device (enp1s0): released from master device bond0
Mar 16 09:13:12 rhvh445-01 NetworkManager[1490]: <info>  [1647421992.9431] device (enp1s0): disconnecting for new activation request.
Mar 16 09:13:12 rhvh445-01 NetworkManager[1490]: <info>  [1647421992.9432] device (enp8s0): state change: activated -> deactivating (reason 'new-activation', sys-iface-state: 'managed')
Mar 16 09:13:12 rhvh445-01 kernel: bond0: (slave enp8s0): Releasing backup interface
Mar 16 09:13:12 rhvh445-01 kernel: device enp8s0 left promiscuous mode
Mar 16 09:13:12 rhvh445-01 NetworkManager[1490]: <info>  [1647421992.9526] device (bond0): released bond slave enp8s0
Mar 16 09:13:12 rhvh445-01 kernel: test1: port 1(bond0.20) entered disabled state
Mar 16 09:13:12 rhvh445-01 kernel: 8021q: adding VLAN 0 to HW filter on device enp8s0
Mar 16 09:13:12 rhvh445-01 NetworkManager[1490]: <info>  [1647421992.9559] device (bond0.20): set-hw-addr: set MAC address to 52:0F:96:CE:BF:D3 (vlan-parent)
Mar 16 09:13:12 rhvh445-01 NetworkManager[1490]: <info>  [1647421992.9564] device (bond0.20): set-hw-addr: set MAC address to 52:54:00:18:95:CE (vlan-parent)
Mar 16 09:13:12 rhvh445-01 NetworkManager[1490]: <info>  [1647421992.9564] device (bond0): set-hw-addr: set MAC address to 52:54:00:18:95:CE (restore)
Mar 16 09:13:12 rhvh445-01 NetworkManager[1490]: <info>  [1647421992.9576] device (enp8s0): released from master device bond0
Mar 16 09:13:12 rhvh445-01 NetworkManager[1490]: <info>  [1647421992.9582] device (enp8s0): disconnecting for new activation request.
Mar 16 09:13:12 rhvh445-01 NetworkManager[1490]: <info>  [1647421992.9598] device (bond0): state change: deactivating -> disconnected (reason 'new-activation', sys-iface-state: 'managed')


Expected results:
The interface is not brought down. IMO autoconnect-slaves is not an actual setting that needs to be applied to the interface, but more of a management setting in NetworkManager's internal state for the connection.

Additional info:
Comment 1 Thomas Haller 2022-03-17 08:55:40 UTC 
> The interface is not brought down. IMO autoconnect-slaves is not an actual setting that needs to be applied to the interface, but more of a management setting in NetworkManager's internal state for the connection.

I agree.

https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/1150
Comment 2 Thomas Haller 2022-03-18 14:37:40 UTC 
fixed upstream (https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/1150)
Comment 5 Matej Berezny 2022-04-20 22:07:56 UTC 
updated tests to check for this as well (https://gitlab.freedesktop.org/NetworkManager/NetworkManager-ci/-/merge_requests/1025)
Comment 7 errata-xmlrpc 2022-11-08 10:07:32 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (NetworkManager bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:7680
Note You need to log in before you can comment on or make changes to this bug.