RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1932247 - Veths attached to nmstate controlled bridge become managed
Summary: Veths attached to nmstate controlled bridge become managed
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: nmstate
Version: 8.3
Hardware: Unspecified
OS: Unspecified
unspecified
urgent
Target Milestone: rc
: ---
Assignee: Fernando F. Mancera
QA Contact: Mingyu Shi
URL:
Whiteboard:
Depends On:
Blocks: 1931376 1932972 1936432
TreeView+ depends on / blocked
 
Reported: 2021-02-24 11:08 UTC by Petr Horáček
Modified: 2021-05-18 15:18 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1932972 (view as bug list)
Environment:
Last Closed: 2021-05-18 15:18:24 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
pre-tested.log (70.67 KB, text/plain)
2021-03-04 05:48 UTC, Mingyu Shi 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Github nmstate nmstate pull 1220/commits/85a1c34e72f47cf6fb555ce6d1c71d677133ea98 0 None None None 2021-02-24 13:24:25 UTC
Red Hat Bugzilla 1855189 1 medium CLOSED Fail to change bridge slave if an unmanaged slave exists 2023-06-26 12:33:15 UTC

Description Petr Horáček 2021-02-24 11:08:05 UTC 
Description of problem:

Starting with nmstate 0.3, any veth iface attached to a bridge created through nmstate becomes managed when the configuration of the bridge gets reapplied. This happens despite there are udev rules making all veths unmanaged.

Due to this, VM's that get attached to a Linux bridge in OpenShift Virtualization get disconnected as soon as nmstate reapplies the config (on configuration change or mere restart of knmstate).

This was not the case in nmstate 0.2.

Version-Release number of selected component (if applicable):
nmstate 0.3.4
NetworkManager 1.26.0

How reproducible:
Always.


Steps to Reproduce:

    echo 'interfaces:
    - bridge:
        options:
        stp:
            enabled: false
    ipv4:
        dhcp: false
        enabled: false
    ipv6:
        enabled: false
    name: upg-br-mark
    state: up
    type: linux-bridge
    ' > br.yaml

    nmstatectl set br.yaml

    ip link add veth23da3115 type veth peer name veth_x
    ip netns add test
    ip link set veth_x netns test
    ip link set veth23da3115 master br1
    ip link | grep br1

    nmcli d | grep veth23da3115
    # veth23da3115    ethernet       unmanaged     --

    nmstatectl set br.yaml

    # With nmstate 0.3 it becomes managed
    nmcli d | grep veth23da3115
    # veth23da3115    ethernet       connected     veth23da3115

    # With nmstate 0.2 it stays unmanaged
    nmcli d | grep veth23da3115
    # veth23da3115    ethernet       unmanaged     --


Actual results:
Veth becomes managed and then can be randomly disconnected by NetworkManager.


Expected results:
Veths should remain unmanaged.

Additional notes:

This udev rule seems to be ignored:

[root@network06-nwrdz-worker-0-lf2mw core]# cat /usr/lib/udev/rules.d/85-nm-unmanaged.rules
...

# Virtual Ethernet device pair. Often used to communicate with a peer interface
# in another net namespace and managed by libvirt, Docker or the like.
ENV{ID_NET_DRIVER}=="veth", ENV{NM_UNMANAGED}="1"

...
Comment 1 Petr Horáček 2021-02-24 12:02:18 UTC 
Note that unlike the udev rule, applying echo "unmanaged-devices=interface-name:veth*" NetworkManager configuration explicitly works around the issue.
Comment 2 Thomas Haller 2021-02-24 12:33:58 UTC 
Making  device unmanaged via udev rule (or "device.managed" setting in NetworkManager.conf), can be overruled by the user.

On the other hand, "keyfile.unmanaged-devices" in NetworkManager.conf and NM_CONTROLLED=no in ifcfg cannot be overruled. The device is always unmanaged.

But the solution here is not to use "keyfile.unmanaged-devices".

The solution is to see who tells NetworkManager to use this device (and, if that is undesirable, prevent it from doing that).
Comment 3 Fernando F. Mancera 2021-02-24 12:37:15 UTC 
(In reply to Thomas Haller from comment #2)
> Making  device unmanaged via udev rule (or "device.managed" setting in
> NetworkManager.conf), can be overruled by the user.
> 
> On the other hand, "keyfile.unmanaged-devices" in NetworkManager.conf and
> NM_CONTROLLED=no in ifcfg cannot be overruled. The device is always
> unmanaged.
> 
> But the solution here is not to use "keyfile.unmanaged-devices".
> 
> The solution is to see who tells NetworkManager to use this device (and, if
> that is undesirable, prevent it from doing that).

I can confirm that Nmstate is telling NetworkManager to use that device. I am looking how it would be possible to prevent this.
Comment 4 Fernando F. Mancera 2021-02-24 12:42:25 UTC 
(In reply to Fernando F. Mancera from comment #3)
> (In reply to Thomas Haller from comment #2)
> > Making  device unmanaged via udev rule (or "device.managed" setting in
> > NetworkManager.conf), can be overruled by the user.
> > 
> > On the other hand, "keyfile.unmanaged-devices" in NetworkManager.conf and
> > NM_CONTROLLED=no in ifcfg cannot be overruled. The device is always
> > unmanaged.
> > 
> > But the solution here is not to use "keyfile.unmanaged-devices".
> > 
> > The solution is to see who tells NetworkManager to use this device (and, if
> > that is undesirable, prevent it from doing that).
> 
> I can confirm that Nmstate is telling NetworkManager to use that device. I
> am looking how it would be possible to prevent this.

Is the reproducer complete or are there some nmstate commands missing?
Comment 5 Thomas Haller 2021-02-24 12:56:21 UTC 
I mean, if you really want, you can also configure "keyfile.unmanaged-devices" to force the device as being unmanaged.

In general that behavior is not very useful, because it means an explicit user's request like `nmcli device set $IFNAME managed yes` and `nmcli device connect $IFNAME` will fail. If the user doesn't want NetworkManager to manage a device, then it suffices to set the udev rule (or "device.managed=0") and simply not tell NetworkManager to manage it.

Hence, it would be better to investigate why nmstate wants to manage the device and don't do that.
Comment 6 Fernando F. Mancera 2021-02-24 13:22:53 UTC 
I've found why Nmstate is setting the port as managed. The commit causing this is https://github.com/nmstate/nmstate/pull/1220/commits/85a1c34e72f47cf6fb555ce6d1c71d677133ea98 which solves: https://bugzilla.redhat.com/show_bug.cgi?id=1855189

Reverting this commit will fix this issue but will break the existing one. The solution will be to see if the unmanaged device is up or down before creating the profile, but on nmstate-0.3 that is not possible if the interface is unmanaged because we are using NetworkManager to query the network state.
Comment 7 Fernando F. Mancera 2021-02-24 13:24:26 UTC 
(In reply to Fernando F. Mancera from comment #6)
> I've found why Nmstate is setting the port as managed. The commit causing
> this is
> https://github.com/nmstate/nmstate/pull/1220/commits/
> 85a1c34e72f47cf6fb555ce6d1c71d677133ea98 which solves:
> https://bugzilla.redhat.com/show_bug.cgi?id=1855189
> 
> Reverting this commit will fix this issue but will break the existing one.

I mean the one already fixed.

> The solution will be to see if the unmanaged device is up or down before
> creating the profile, but on nmstate-0.3 that is not possible if the
> interface is unmanaged because we are using NetworkManager to query the
> network state.
Comment 8 Fernando F. Mancera 2021-02-24 22:49:04 UTC 
Upstream PR: https://github.com/nmstate/nmstate/pull/1529
Comment 12 Petr Horáček 2021-02-27 18:37:02 UTC 
The patch failed our verification. While it works fine when the bridge has no physical NIC (managed) attached, when the bridge is configured on top of eth0, all veths get disconnected on apply. My original U/S verification tried only the former, but we failed our D/S tests on the latter.
Comment 13 Gris Ge 2021-03-02 06:01:11 UTC 
RHEL 8.4 patch posted upstream: https://github.com/nmstate/nmstate/pull/1531
Comment 14 Mingyu Shi 2021-03-04 05:48:16 UTC 
Created attachment 1760611 [details]
pre-tested.log

Tested:
nmstate-1.0.3-0.20210302073301710589.pr1531.6.g3e5bfca.el8.noarch
nispor-1.0.1-4.el8.x86_64
NetworkManager-1.30.0-2.el8.x86_64
Comment 17 Mingyu Shi 2021-03-08 04:04:51 UTC 
Verified with versions:
nmstate-1.0.2-5.el8.noarch
nispor-1.0.1-4.el8.x86_64
NetworkManager-1.30.0-2.el8.x86_64

For bond, vrf and linux-bridge, their unmanaged subordinate ports will keep as unmanged until you explicitly list these ports under 'interfaces' key.
Currently team and ovs don't full support if itself or its subordinate port is not managed by NM.
Comment 19 errata-xmlrpc 2021-05-18 15:18:24 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (nmstate bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1748
Note You need to log in before you can comment on or make changes to this bug.