This bug has been migrated to another issue tracking site. It has been closed here and may no longer be being monitored.

If you would like to get updates for this issue, or to participate in it, you may do so at Red Hat Issue Tracker .
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2161994 - kernel wrongly merges IPv6 ECMP routes with next-hop with append/prepend
Summary: kernel wrongly merges IPv6 ECMP routes with next-hop with append/prepend
Keywords:
Status: CLOSED MIGRATED
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: kernel
Version: 9.2
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: Hangbin Liu
QA Contact: Jianlin Shi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2023-01-18 14:25 UTC by Thomas Haller
Modified: 2023-09-21 12:57 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2023-09-21 12:57:18 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
reproducer using iproute2 showing how routes get merged (6.22 KB, text/plain)
2023-01-18 14:27 UTC, Thomas Haller 		no flags Details
output of reproducer using iproute2 showing how routes get merged (19.45 KB, text/plain)
2023-01-18 14:28 UTC, Thomas Haller 		no flags Details
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker NMT-118 0 None None None 2023-01-22 14:16:49 UTC
Red Hat Issue Tracker   RHEL-6048 0 None Migrated None 2023-09-21 12:57:13 UTC
Red Hat Issue Tracker RHELPLAN-145550 0 None None None 2023-01-18 14:29:56 UTC

Description Thomas Haller 2023-01-18 14:25:59 UTC 
With `ip -6 route append|prepend`, kernel wrongly merges routes that have a next-hop set.

Actually, the kernel behavior is rather confusing (maybe even inconsistent and nonsensical?). It's hard to point out the single wrong thing with it. Let's just look at the two following examples, maybe if we understand/fix those, the broader problem is fixed/understandable:

Example 1:

  # ip -6 route append unicast 1:2:3::6 dev v proto kernel table 5
  # ip -6 route append blackhole 1:2:3::6 dev v proto kernel table 5
  # ip -6 route append local 1:2:3::6 dev v proto kernel table 5
  RTNETLINK answers: File exists

or the other way around:

  # ip -6 route append local 1:2:3::6 dev v proto kernel table 5
  # ip -6 route append unicast 1:2:3::6 dev v proto kernel table 5
  RTNETLINK answers: File exists

So we can add/append two routes that only differ by their type ("unicast" vs. "blackhole"), but not if the type is "local".

The question is "what is a route?" (bug 1337855). A route (or policy routing rule) in kernel doesn't have a small set of attributes that make up an identity of the route. Instead, there is a larger set of attributes, that determine what a route "is". As seen above, we can configure two routes that only differ by their type ("unicast" and "blackhole"). That suggest that the type is part
of the identity of a route.

Note that altought the type is part of the identity of a route, we cannot add a type "local" route together with a type "unicast" route. OK, that may not yet invalidate the idea that the route type is part of the identity, it just means that despite a "local" route and a "unicast" route being different routes (all other attributes being equal), they cannot be configured at the same time. Kernel has various such inter-dependencies for other cases (e.g. when there is a next hop, there must be another route so that the gateway is reachable on-link).


But see the attached reproducer... it behaves similar on Fedora 37 (6.0.18-300.fc37.x86_64) and rhel-9.2 (5.14.0-234.el9.x86_64). The only difference is that the `ip monitor` on rhel-9.2 doesn't output anything. Dunno why. Another bug? I'll attach the output of the script from Fedora 37.

That script falls out form a unit test of NetworkManager, where NetworkManager configures randomly some routes and eventually it hits the problem. In this case, the problem is if routes have a next-hop. IPv6 routes with a next-hop are automatically merged by kernel. So in the script we see first that
we have a local route:

  > local a:b:c:f::/64 via 7:7:7:7::1 dev net0 table 10222 proto kernel scope global metric 20 pref medium

then we prepend a unicast route:

  # ip route prepend a:b:c:f::/64 proto bgp metric 20 table 10222 nexthop via 7:7:7:7::1 dev net1

we get a netlink notification for a unicast route (on Fedora only):

  > a:b:c:f::/64 table 10222 proto bgp metric 20 pref medium
  >     nexthop via 7:7:7:7::1 dev net1 weight 1·
  >     nexthop via 7:7:7:7::1 dev net0 weight 5·

and afterwards `ip route show` will give:

  >  local a:b:c:f::/64 table 10222 proto kernel scope global metric 20 pref medium
  >      nexthop via 7:7:7:7::1 dev net0 weight 5·
  >      nexthop via 7:7:7:7::1 dev net1 weight 1·


So the merging seems really wrong. The netlink notification does also not match to what is happening (or doesn't come on rhel-9.2?).

I think the problem is related to how "type local" routes are treated in the example above. The solution is possible to ensure that route with type "local" don't conflict/overlap/merge with "unicast" routes.



The unit test in NetworkManager can probably uncover other scenarios. But let's address the attached reproducer first, and see afterwards whether other
issues can be uncovered beyond that. Please reach out how to run NetworkManager's unit tests (it is currently patched to not cover the problematic case).


This is a severe problem, because

1) adding a "unicast" route will be merged with a "local" route, which seems very wrong.

2) the netlink events are not correct. NetworkManager maintains a cache of the routes. It requires that the netlink events allow to find out what is happening. The wrong netlink notification makes it impossible for NetworkManager to understand what happened. Netlink events are already insufficient with `ip route replace` (bug 1337860), so when NetworkManager sees an RTM_NEWROUTE with NLM_F_REPLACE flags, it already requests already new dump of all routes to resync -- which is expensive. But in this case, NetworkManager cannot tell from the netlink notification that something odd happend, which would require a re-sync.



PS: I still have hope that one day bug 1337855 and bug 1337860 could be resolved.
Comment 1 Thomas Haller 2023-01-18 14:27:38 UTC 
Created attachment 1938904 [details]
reproducer using iproute2 showing how routes get merged
Comment 2 Thomas Haller 2023-01-18 14:28:31 UTC 
Created attachment 1938905 [details]
output of reproducer using iproute2 showing how routes get merged

Run on Fedora 37. Similar on rhel-9.2 (but `ip monitor` outputs nothing there).
Comment 3 Thomas Haller 2023-01-19 11:05:06 UTC 
btw, the mentioned unit test of NetworkManager is at [1].

[1] https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/blob/adad1d435814e6e195c97836073bce62bac40a47/src/core/platform/tests/test-route.c#L2280

By disabling the "if", the test starts failing and reproduces the issue.

It can be run with:

  NMTST_DEBUG=d,slow ./tools/run-nm-test.sh -m src/core/platform/tests/test-route-linux

Please reach out if you try to do that.
Comment 4 Thomas Haller 2023-04-04 12:49:44 UTC 
reassigning to kernel. This is about how kernel merges routes IPv6 ECMP routes.
Comment 5 Hangbin Liu 2023-07-26 09:20:21 UTC 
(In reply to Thomas Haller from comment #0)
> With `ip -6 route append|prepend`, kernel wrongly merges routes that have a
> next-hop set.
> 
> Actually, the kernel behavior is rather confusing (maybe even inconsistent
> and nonsensical?). It's hard to point out the single wrong thing with it.
> Let's just look at the two following examples, maybe if we understand/fix
> those, the broader problem is fixed/understandable:
> 
> Example 1:
> 
>   # ip -6 route append unicast 1:2:3::6 dev v proto kernel table 5
>   # ip -6 route append blackhole 1:2:3::6 dev v proto kernel table 5
>   # ip -6 route append local 1:2:3::6 dev v proto kernel table 5
>   RTNETLINK answers: File exists
> 
> or the other way around:
> 
>   # ip -6 route append local 1:2:3::6 dev v proto kernel table 5
>   # ip -6 route append unicast 1:2:3::6 dev v proto kernel table 5
>   RTNETLINK answers: File exists


In kernel function rtm_to_fib6_config(), if the route type is RTN_UNREACHABLE/RTN_BLACKHOLE/RTN_PROHIBIT/RTN_THROW.
The kernel will add RTF_REJECT flag. Then in fib6_nh_init() kernel set dev to lo. So they are treated as different routes.

# ip -6 route show table 5
1:2:3::6 dev dummy0 metric 1024 pref medium
blackhole 1:2:3::6 dev lo metric 1024 pref medium

But local/unicast routes are checked as same routes as they have the same dest and device.

> 
>   > local a:b:c:f::/64 via 7:7:7:7::1 dev net0 table 10222 proto kernel
> scope global metric 20 pref medium
> 
> then we prepend a unicast route:
> 
>   # ip route prepend a:b:c:f::/64 proto bgp metric 20 table 10222 nexthop
> via 7:7:7:7::1 dev net1

When IPv6 checks the routes, it only checks the dest address and devices.
The type and protocol fields are ignored. If the dest is same while device is different.
They will be merged to multihost routes.

> 
> we get a netlink notification for a unicast route (on Fedora only):
> 
>   > a:b:c:f::/64 table 10222 proto bgp metric 20 pref medium
>   >     nexthop via 7:7:7:7::1 dev net1 weight 1·
>   >     nexthop via 7:7:7:7::1 dev net0 weight 5·

The reason why you got proto bgp is that when dumping the route, the kernel use
the configs from user space(actually the type and protocol are merged to the first one and ignored at latest).

> 
> and afterwards `ip route show` will give:
> 
>   >  local a:b:c:f::/64 table 10222 proto kernel scope global metric 20 pref
> medium
>   >      nexthop via 7:7:7:7::1 dev net0 weight 5·
>   >      nexthop via 7:7:7:7::1 dev net1 weight 1·
> 

So when really dump the route, The type and protocol are used the first hops info.

> 
> This is a severe problem, because
> 
> 1) adding a "unicast" route will be merged with a "local" route, which seems
> very wrong.
> 
> 2) the netlink events are not correct. NetworkManager maintains a cache of
> the routes. It requires that the netlink events allow to find out what is
> happening. The wrong netlink notification makes it impossible for
> NetworkManager to understand what happened. Netlink events are already
> insufficient with `ip route replace` (bug 1337860), so when NetworkManager
> sees an RTM_NEWROUTE with NLM_F_REPLACE flags, it already requests already
> new dump of all routes to resync -- which is expensive. But in this case,
> NetworkManager cannot tell from the netlink notification that something odd
> happend, which would require a re-sync.
Comment 6 Thomas Haller 2023-07-26 09:46:45 UTC 
> When IPv6 checks the routes, it only checks the dest address and devices.
The type and protocol fields are ignored.
> ...
> So when really dump the route, The type and protocol are used the first hops info.

Yes, I know :)

I claim that that is a bug and it shouldn't be like that.
Comment 7 Hangbin Liu 2023-08-02 06:23:43 UTC 
(In reply to Thomas Haller from comment #6)
> > When IPv6 checks the routes, it only checks the dest address and devices.
> The type and protocol fields are ignored.
> > ...
> > So when really dump the route, The type and protocol are used the first hops info.
> 
> Yes, I know :)
> 
> I claim that that is a bug and it shouldn't be like that.

As upstream suggested, it's hard to do the change on legacy nexthop api and the new nexthop api should resolve this issue. e.g.

# ip addr add 2001:db8:101::1/64 dev dummy1
# ip addr add 2001:db8:101::2/64 dev dummy2
# ip nexthop add id 1 via 2001:db8:101::10 dev dummy1
# ip nexthop add id 2 via 2001:db8:101::10 dev dummy2
# ip nexthop add id 101 group 1/2
# ip nexthop show
id 1 via 2001:db8:101::10 dev dummy1 scope link
id 2 via 2001:db8:101::10 dev dummy2 scope link
id 101 group 1/2

# ip route add table 100 2001:db8:102::/64 nhid 1
# ip route prepend table 100 2001:db8:102::/64 nhid 1
RTNETLINK answers: File exists
# ip route add table 100 2001:db8:102::/64 nhid 2
RTNETLINK answers: File exists
# ip route prepend table 100 2001:db8:102::/64 nhid 2
# ip -6 route show table 100                                                      <- not merged
2001:db8:102::/64 nhid 1 via 2001:db8:101::10 dev dummy1 metric 1024 pref medium
2001:db8:102::/64 nhid 2 via 2001:db8:101::10 dev dummy2 metric 1024 pref medium

# ip route add table 100 local 2001:db8:103::/64 nhid 1
# ip route prepend table 100 unicast 2001:db8:103::/64 nhid 1
RTNETLINK answers: File exists
# ip route prepend table 100 unicast 2001:db8:103::/64 nhid 2
# ip -6 route show table 100                                                      <- local/unicast are separated
local 2001:db8:103::/64 nhid 1 via 2001:db8:101::10 dev dummy1 metric 1024 pref medium
2001:db8:103::/64 nhid 2 via 2001:db8:101::10 dev dummy2 metric 1024 pref medium

# ip route add table 100 2001:db8:104::/64 nhid 1 proto kernel
# ip route prepend table 100 2001:db8:104::/64 nhid 1 proto bgp
RTNETLINK answers: File exists
# ip route prepend table 100 2001:db8:104::/64 nhid 2 proto bgp
# ip -6 route show table 100                                                      <- kernel/bgp are separated
2001:db8:104::/64 nhid 1 via 2001:db8:101::10 dev dummy1 proto kernel metric 1024 pref medium
2001:db8:104::/64 nhid 2 via 2001:db8:101::10 dev dummy2 proto bgp metric 1024 pref medium

# ip -6 route add table 100 local 2001:db8:105::/64 nhid 101 proto kernel
# ip -6 route show table 100
local 2001:db8:105::/64 nhid 101 proto kernel metric 1024 pref medium
        nexthop via 2001:db8:101::10 dev dummy1 weight 1
        nexthop via 2001:db8:101::10 dev dummy2 weight 1
# ip -6 route prepend table 100 unicast 2001:db8:105::/64 nhid 101 proto bgp
RTNETLINK answers: File exists
Comment 8 Thomas Haller 2023-08-02 07:24:58 UTC 
> it's hard to do the change on legacy nexthop api and the new nexthop api should resolve this issue. e.g.

Is that relevant?

Even if NetworkManager uses nexthops and never configures such a thing itself.

Any user can type such commands in the terminal. Kernel will do the wrong thing (of mangling existing routes). Kernel will also send wrong netlink notifictions, confusing anybody who cares about those notifications (e.g. NetworkManager).
Comment 9 RHEL Program Management 2023-09-21 12:57:04 UTC 
Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.
Comment 10 RHEL Program Management 2023-09-21 12:57:18 UTC 
This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.
Note You need to log in before you can comment on or make changes to this bug.