RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2218866 - 9.3 regression: NetworkManager restores src route after ip route replace/change
Summary: 9.3 regression: NetworkManager restores src route after ip route replace/change
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: NetworkManager
Version: 9.3
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: 9.3
Assignee: Beniamino Galvani
QA Contact: Desktop QE
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-06-30 10:23 UTC by Oyvind Albrigtsen
Modified: 2023-09-21 15:54 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-08-03 16:14:21 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
first try of reproducer (1.67 KB, application/x-shellscript)
2023-07-20 06:27 UTC, Gris Ge 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker NMT-646 0 None None None 2023-06-30 10:25:35 UTC
Red Hat Issue Tracker RHELPLAN-161315 0 None None None 2023-06-30 10:25:40 UTC
freedesktop.org Gitlab NetworkManager NetworkManager-ci merge_requests 1463 0 None opened ipv4: check that pre-1.43 route behavior is restored 2023-08-01 10:35:27 UTC
freedesktop.org Gitlab NetworkManager NetworkManager merge_requests 1696 0 None opened Revert "platform: always reconfigure IP routes even if removed externally" 2023-08-01 10:35:29 UTC

Description Oyvind Albrigtsen 2023-06-30 10:23:28 UTC 
Description of problem:
NetworkManager restores source route for non-NetworkManager managed IPs in 9.3.

This causes issues for our IPsrcaddr resource agent (for Pacemaker High Availability cluster), as it runs "ip route replace" and "ip route change", and then expects there to be only 1 match for the route for it's monitor and stop-actions.

This is not an issue in 9.2 or earlier.

It might be reproducible by simply doing "ip addr add" for the IP followed by "ip route replace", but we've only tested it with the IPaddr2 and IPsrcaddr resource agents.

[root@virt-029 ~]# ip route
default via 10.37.167.254 dev ens3 proto dhcp src 10.37.166.156 metric 100
10.37.164.0/22 dev ens3 proto kernel scope link src 10.37.166.156 metric 100
192.168.2.0/23 dev ens5 proto kernel scope link src 192.168.2.29 metric 101
[root@virt-029 ~]#
 
 
[root@virt-029 ~]# date; pcs resource debug-start OutboundIP
Thu Jun 29 04:55:54 PM CEST 2023
Operation force-start for OutboundIP (ocf:heartbeat:IPsrcaddr) returned 0 (ok)
 
 
Jun 29 16:55:55 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050555.5430] platform: (ens3) signal: route   4   added: type unicast 10.37.164.0/22 dev 2 metric 100 mss 0 rt-src rt-kernel scope link pref-src 10.37.165.104
Jun 29 16:55:55 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050555.5431] platform: (ens3) signal: route   4 removed: type unicast 10.37.164.0/22 dev 2 metric 100 mss 0 rt-src rt-kernel scope link pref-src 10.37.166.156
Jun 29 16:55:55 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050555.5431] l3cfg[97c1f3068a48f490,ifindex=2]: obj-state: remove from platform: [eef54b93eab1dcce, ip4-route, type unicast 10.37.164.0/22 dev 2 metric 100 mss 0 rt-src rt-kernel scope link pref-src 10.37.166.156], nm-configured, was-in-platform
Jun 29 16:55:55 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050555.5441] platform: (ens3) signal: route   4   added: type unicast 0.0.0.0/0 via 10.37.167.254 dev 2 metric 100 mss 0 rt-src rt-dhcp scope global pref-src 10.37.165.104
Jun 29 16:55:55 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050555.5441] platform: (ens3) signal: route   4 removed: type unicast 0.0.0.0/0 via 10.37.167.254 dev 2 metric 100 mss 0 rt-src rt-dhcp scope global pref-src 10.37.166.156
Jun 29 16:55:55 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050555.5442] l3cfg[97c1f3068a48f490,ifindex=2]: obj-state: remove from platform: [4706c32d1c6629ef, ip4-route, type unicast 0.0.0.0/0 via 10.37.167.254 dev 2 metric 100 mss 0 rt-src dhcp pref-src 10.37.166.156], nm-configured, was-in-platform
...
...Jun 29 16:57:25 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050645.0717] ndisc-lndp[0x5645fdbe0610,"ens3"]: processing libndp events
Jun 29 16:57:25 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050645.0719] ndisc-lndp[0x5645fdbe0610,"ens3"]: received router advertisement at timestamp 727.539 seconds
Jun 29 16:57:25 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050645.0720] ndisc[0x5645fdbe0610,"ens3"]: router-data: next lifetime expiration will happen: in 1680.700 seconds
Jun 29 16:57:25 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050645.0720] ndisc[0x5645fdbe0610,"ens3"]: neighbor discovery configuration changed [GAR]:
Jun 29 16:57:25 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050645.0720] ndisc[0x5645fdbe0610,"ens3"]:   dhcp-level none
Jun 29 16:57:25 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050645.0720] ndisc[0x5645fdbe0610,"ens3"]:   hop limit      : 64
Jun 29 16:57:25 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050645.0720] ndisc[0x5645fdbe0610,"ens3"]:   gateway fe80::200:5eff:fe00:201 pref medium exp 1681.092
Jun 29 16:57:25 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050645.0721] ndisc[0x5645fdbe0610,"ens3"]:   gateway fe80::200:5eff:fe00:202 pref medium exp 1680.985
Jun 29 16:57:25 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050645.0721] ndisc[0x5645fdbe0610,"ens3"]:   gateway fe80::82ac:ac00:9e5e:3661 pref medium exp 1680.700
Jun 29 16:57:25 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050645.0721] ndisc[0x5645fdbe0610,"ens3"]:   gateway fe80::327c:5e00:9e8c:7c61 pref medium exp 1800.000
Jun 29 16:57:25 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050645.0721] ndisc[0x5645fdbe0610,"ens3"]:   address 2620:52:0:25a4:1800:ff:fe00:1d exp 2592000.000
Jun 29 16:57:25 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050645.0721] ndisc[0x5645fdbe0610,"ens3"]:   route 2620:52:0:25a4::/64 via :: pref medium exp 2592000.000
Jun 29 16:57:25 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050645.0723] ndisc[0x5645fdbe0610,"ens3"]: solicit: schedule sending next (slow) solicitation in about 1623.264 seconds
Jun 29 16:57:25 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050645.0723] ndisc-lndp[0x5645fdbe0750,"ens5"]: processing libndp events
Jun 29 16:57:25 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050645.0724] l3cfg[97c1f3068a48f490,ifindex=2]: obj-state: update: [931dea6a98b28ecf, ip6-address, 2620:52:0:25a4:1800:ff:fe00:1d/64 lft 2592001sec pref 604801sec lifetime 727-727[604801,2592001] dev 2 flags noprefixroute src ndisc], nm-configured, in-platform
Jun 29 16:57:25 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050645.0724] dns-mgr: (device_l3cd_changed): queueing DNS updates (1)
Jun 29 16:57:25 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050645.0725] dns-mgr: (device_l3cd_changed): DNS configuration did not change
Jun 29 16:57:25 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050645.0725] dns-mgr: (device_l3cd_changed): no DNS changes to commit (0)
Jun 29 16:57:25 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050645.0726] platform: (ens3) route: append     IPv4 route: type unicast 10.37.164.0/22 dev 2 metric 100 mss 0 rt-src rt-kernel scope link pref-src 10.37.166.156
Jun 29 16:57:25 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050645.0727] platform: (ens3) signal: route   4   added: type unicast 10.37.164.0/22 dev 2 metric 100 mss 0 rt-src rt-kernel scope link pref-src 10.37.166.156
Jun 29 16:57:25 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050645.0727] l3cfg[97c1f3068a48f490,ifindex=2]: obj-state: appeared in platform: [eef54b93eab1dcce, ip4-route, type unicast 10.37.164.0/22 dev 2 metric 100 mss 0 rt-src rt-kernel scope link pref-src 10.37.166.156], nm-configured, in-platform
Jun 29 16:57:25 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050645.0728] platform-linux: do-add-ip4-route[type unicast 10.37.164.0/22 dev 2 metric 100 mss 0 rt-src rt-kernel scope link pref-src 10.37.166.156]: success
Jun 29 16:57:25 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050645.0728] platform: (ens3) route: append     IPv4 route: type unicast 0.0.0.0/0 via 10.37.167.254 dev 2 metric 100 mss 0 rt-src rt-dhcp scope global pref-src 10.37.166.156
Jun 29 16:57:25 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050645.0728] platform: (ens3) signal: route   4   added: type unicast 0.0.0.0/0 via 10.37.167.254 dev 2 metric 100 mss 0 rt-src rt-dhcp scope global pref-src 10.37.166.156
Jun 29 16:57:25 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050645.0728] l3cfg[97c1f3068a48f490,ifindex=2]: obj-state: appeared in platform: [4706c32d1c6629ef, ip4-route, type unicast 0.0.0.0/0 via 10.37.167.254 dev 2 metric 100 mss 0 rt-src dhcp pref-src 10.37.166.156], nm-configured, in-platform
Jun 29 16:57:25 virt-029.cluster-qe.lab.eng.brq.redhat.com NetworkManager[6919]: <debug> [1688050645.0728] platform-linux: do-add-ip4-route[type unicast 0.0.0.0/0 via 10.37.167.254 dev 2 metric 100 mss 0 rt-src rt-dhcp scope global pref-src 10.37.166.156]: success
 
 
[root@virt-029 ~]# ip route
default via 10.37.167.254 dev ens3 proto dhcp src 10.37.165.104 metric 100
default via 10.37.167.254 dev ens3 proto dhcp src 10.37.166.156 metric 100
10.37.164.0/22 dev ens3 proto kernel scope link src 10.37.165.104 metric 100
10.37.164.0/22 dev ens3 proto kernel scope link src 10.37.166.156 metric 100
192.168.2.0/23 dev ens5 proto kernel scope link src 192.168.2.29 metric 101

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. Run "ip route replace" for IP not managed by NetworkManager.
2.
3.

Actual results:
Old and new src IPs present in routing table.

Expected results:
Only new src IP present.

Additional info:
replace/change commands from the IPsrcaddr resource agent:
+ 14:19:20: srca_start:252: ip route replace 10.37.164.0/22 dev ens3 proto kernel src 10.37.165.110 metric 100
+ 14:19:20: srca_start:256: ip route change to default via 10.37.167.254 proto dhcp metric 100 src 10.37.165.110
Comment 1 Thomas Haller 2023-06-30 13:41:33 UTC 
in the past, NetworkManager made an effort to not restore a route, when it was removed externally.

However, that causes problems and was "fixed" (well, it's not a fix from POV of the reporter).

- when a route disappears, it's not clear why it's gone and whether that is what the user desired. For example, if you have a route with "prefsrc $IPADDR" and you delete the IP address "$IPADDR", kernel will automatically remove the route too. So far so good, at first NetworkManager cannot re-add the route unless somebody/NetworkManager re-adds $IPADDR address. However, later when the address gets re-added (either by the user or by NM), then NM probably(?) should restore that route. As we don't know whether the route was removed intentionally be the user or as a side effect from something else, we don't know whether we should restore the route. NM now consistently restores it.

- routes depend on each other. For example, a route with a next hop, can only be configured in kernel, if there is also a route that makes the next-hop directly reachable. If the user externally removes such a direct route, NM is unable to configure other routes. It's difficult to keep track of what NM should configure vs. what it can currently configure.

- during a restart of NetworkManager, an interface might only be half configured (if it was in the process of activating). Upon restart, NetworkManager doesn't know why some IP addresses/routes are missing. And it needs to re-configure them.


NetworkManager does not configure the interface all the time, but whenever something happens that triggers a reconfiguration (e.g. a DHCP lease update, a IPv6 Router Adv, a `nmcli device reapply`), NetworkManager will aim to configure all IP addresses and routes, that it planned to configure. That is, it no longer tries to keep track of addresses/routes which it previously configured and which somehow got externally lost (and blacklist them from being configured).

What NetworkManager still does, ignore all externally present/configured addresses/routes, until `nmcli device reapply|modify` gets called. So you can add them without NetworkManager's knowledge or interference. But you cannot remove addresses that NetworkManager wants to configure.

Your application should work in the face where somebody calls `nmcli device reapply` or `nmcli connection up`. Both would restore the IP configuration that NetworkManager intends to be present. If that breaks your application, then you have a problem. If your application can handle that, then that is not different from NetworkManager at undefined times re configuring what it wants.

I don't think this worked reliably in the past either, because the behavior was ill defined. You probably had to be ready to handle the case that NetworkManager would restore an address/route, that you just deleted. For example, imagine the route comes from DHCP and NM configures a route. You delete the route externally and NM on rhel-9.2 would not restore it. Then the DHCP server crashes, the lease times out and NM forgets about the route. Later DHCP works again, and NM would restore the route again.

Note that NetworkManager will not fight hard to restore the route. If you delete it, NM will only restore it when it thinks it should configure the route again. So you could just delete it then again. That's like in the past, where this also didn't reliably work, and you had to be ready to take that action over and over.

I think a much better solution is to not delete routes that NetworkManager adds, but instead only `ip route append` routes with a lower metric.
Comment 2 Thomas Haller 2023-06-30 13:52:27 UTC 
related commit https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/7ca95cee15b32af2452aaf4a165eb5c634fba132
Comment 3 sfaye 2023-07-04 07:35:53 UTC 
Hi Oyvind, 

From comment 1, it looks like restoring routes when they were removed externally is expected in RHEL-9.3. Does the proposal of not deleting routes that NetworkManager adds but instead use "ip route append" work for you? 

Thanks
Comment 4 Oyvind Albrigtsen 2023-07-05 15:03:56 UTC 
The resource agent logic depends too heavily on the replace/change logic, and that's not something we want to change, as it might lead to all kinds of other issues with or without NM. Also we shouldnt change how these things work in the middle of a release.
Comment 5 Chris Feist 2023-07-05 15:17:43 UTC 
Also, my concern is that we are making a significant change in default behavior of Network manager in a minor release.  This may have the potential to break customer scripts.
Comment 6 sfaye 2023-07-07 08:00:02 UTC 
Hi Oyvind, Chris,

I understand your concerns about the changes to NetworkManager's default behavior in the minor release and how it might affect the resource agent logic.
To address this, a member of our team is going to investigate and propose a solution. We are also considering your suggestions and the potential to revert the changes.
I will let you know in this bz about the progress. Do you have a specific timeline you are working with where you would expect a resolution to this issue? 

Thanks
Comment 7 Beniamino Galvani 2023-07-14 09:00:44 UTC 
Hi Oyvind,

in the bug description I see that the agent is replacing a route received via DHCP:

  ip route change to default via 10.37.167.254 proto dhcp metric 100 src 10.37.165.110

My understanding is that even on RHEL 9.2, NetworkManager re-adds the old route again on the next DHCP lease renewal, and the result is the same route duplication reported in comment 0. How does the agent handle that scenario?

Also, can you attach a full log of NetworkManager from startup?
Comment 8 Oyvind Albrigtsen 2023-07-14 09:07:59 UTC 
The IP was added by IPaddr2 agent (which uses ip addr add), and the issue is new in 9.3. It has worked fine in RHEL7 (and probably earlier) up to 9.2.
Comment 9 Beniamino Galvani 2023-07-14 12:47:03 UTC 
(In reply to Oyvind Albrigtsen from comment #8)
> The IP was added by IPaddr2 agent (which uses ip addr add), and the issue is new in 9.3. It has worked fine in RHEL7 (and probably earlier) up to 9.2.

The change done in 9.3 is to restore routes configured in NM if they are removed externally. From what I understand the agent is adding addresses and routes via 'ip' and not through NM and so I am struggling to understand how the change is affecting your scenario. Please attach a full log since NM startup (and at trace level) so that the issue can be reproduced and investigated.
Comment 11 Gris Ge 2023-07-20 06:27:04 UTC 
Created attachment 1976653 [details]
first try of reproducer

According to previous NM logs, the reporter is trying to add another static IP using the same CIDR retrieved from DHCP and replacing both direct route and default gateway with this new static address as source address.

Using attached reproducer `bug_2218866.sh`:

NetworkManager-1.43.11-32486.copr.13d4d4c35c.el9.x86_64 will add the dhcp route back when it got IPv6 router advertisement.

NetworkManager-1.42.2-6.el9_2 only add DHCP route back on next DHCP request. In the reproducer, we set the DHCP lease to 120seconds, so we see DHCP route back after 60 seconds.

The reason reporter thinks RHEL 9.2 does not have this problem is because they have big DHCP lease time 86400 seconds(24 hours).
Comment 12 Beniamino Galvani 2023-07-20 08:46:38 UTC 
> The reason reporter thinks RHEL 9.2 does not have this problem is because they have big DHCP lease time 86400 seconds(24 hours).

Right, that also matches my analysis and what said in comment 7. On RHEL 9.2 the DHCP route is being restored after 12 hours (on lease renewal), while on 9.3 it's restored when a new IPv6 router advertisement is received after few seconds.

In any case, I think we should restore the old behavior because it's risky to introduce such changes in a minor RHEL release.

At the same time, it seems to me that the issue that the IPsrcaddr resource agent is having on 9.3 is also present on 9.2, albeit with a different frequency. Oyvind, do you how the agent handles the fact that NM restores the route on DHCP renewal (also in 9.2)?
Comment 13 Martin Juricek 2023-07-27 07:18:28 UTC 
(In reply to Beniamino Galvani from comment #12)
> > The reason reporter thinks RHEL 9.2 does not have this problem is because they have big DHCP lease time 86400 seconds(24 hours).

You are right, we reproduced the problem also in RHEL-9.2 after 12+ hours. Also the ocf:heartbeat:IPsrcaddr resource agent man page recommends not to use the agent on dhcp enabled interface. So we ran the tests with dhcp disabled for 20+ hours without any problem.
Comment 14 Beniamino Galvani 2023-07-31 16:19:26 UTC 
(In reply to Martin Juricek from comment #13)
> You are right, we reproduced the problem also in RHEL-9.2 after 12+ hours.
> Also the ocf:heartbeat:IPsrcaddr resource agent man page recommends not to
> use the agent on dhcp enabled interface. So we ran the tests with dhcp
> disabled for 20+ hours without any problem.

In the light of this, do you consider the bz still valid? Is the scenario from comment 0 taken from a test case, or was the problem found in a real use case?
Note You need to log in before you can comment on or make changes to this bug.