Bug 2161994
| Summary: | kernel wrongly merges IPv6 ECMP routes with next-hop with append/prepend | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Thomas Haller <thaller> | ||||||
| Component: | kernel | Assignee: | Hangbin Liu <haliu> | ||||||
| kernel sub component: | Networking | QA Contact: | Jianlin Shi <jishi> | ||||||
| Status: | CLOSED MIGRATED | Docs Contact: | |||||||
| Severity: | unspecified | ||||||||
| Priority: | unspecified | CC: | bgalvani, haliu, jiji, kzhang, lrintel, rkhan, sfaye, sukulkar, till | ||||||
| Version: | 9.2 | Keywords: | MigratedToJIRA, Triaged | ||||||
| Target Milestone: | rc | Flags: | pm-rhel:
mirror+
|
||||||
| Target Release: | --- | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Unspecified | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||
| Doc Text: | Story Points: | --- | |||||||
| Clone Of: | Environment: | ||||||||
| Last Closed: | 2023-09-21 12:57:18 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Attachments: |
|
||||||||
Created attachment 1938904 [details]
reproducer using iproute2 showing how routes get merged
Created attachment 1938905 [details]
output of reproducer using iproute2 showing how routes get merged
Run on Fedora 37. Similar on rhel-9.2 (but `ip monitor` outputs nothing there).
btw, the mentioned unit test of NetworkManager is at [1]. [1] https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/blob/adad1d435814e6e195c97836073bce62bac40a47/src/core/platform/tests/test-route.c#L2280 By disabling the "if", the test starts failing and reproduces the issue. It can be run with: NMTST_DEBUG=d,slow ./tools/run-nm-test.sh -m src/core/platform/tests/test-route-linux Please reach out if you try to do that. reassigning to kernel. This is about how kernel merges routes IPv6 ECMP routes. (In reply to Thomas Haller from comment #0) > With `ip -6 route append|prepend`, kernel wrongly merges routes that have a > next-hop set. > > Actually, the kernel behavior is rather confusing (maybe even inconsistent > and nonsensical?). It's hard to point out the single wrong thing with it. > Let's just look at the two following examples, maybe if we understand/fix > those, the broader problem is fixed/understandable: > > Example 1: > > # ip -6 route append unicast 1:2:3::6 dev v proto kernel table 5 > # ip -6 route append blackhole 1:2:3::6 dev v proto kernel table 5 > # ip -6 route append local 1:2:3::6 dev v proto kernel table 5 > RTNETLINK answers: File exists > > or the other way around: > > # ip -6 route append local 1:2:3::6 dev v proto kernel table 5 > # ip -6 route append unicast 1:2:3::6 dev v proto kernel table 5 > RTNETLINK answers: File exists In kernel function rtm_to_fib6_config(), if the route type is RTN_UNREACHABLE/RTN_BLACKHOLE/RTN_PROHIBIT/RTN_THROW. The kernel will add RTF_REJECT flag. Then in fib6_nh_init() kernel set dev to lo. So they are treated as different routes. # ip -6 route show table 5 1:2:3::6 dev dummy0 metric 1024 pref medium blackhole 1:2:3::6 dev lo metric 1024 pref medium But local/unicast routes are checked as same routes as they have the same dest and device. > > > local a:b:c:f::/64 via 7:7:7:7::1 dev net0 table 10222 proto kernel > scope global metric 20 pref medium > > then we prepend a unicast route: > > # ip route prepend a:b:c:f::/64 proto bgp metric 20 table 10222 nexthop > via 7:7:7:7::1 dev net1 When IPv6 checks the routes, it only checks the dest address and devices. The type and protocol fields are ignored. If the dest is same while device is different. They will be merged to multihost routes. > > we get a netlink notification for a unicast route (on Fedora only): > > > a:b:c:f::/64 table 10222 proto bgp metric 20 pref medium > > nexthop via 7:7:7:7::1 dev net1 weight 1· > > nexthop via 7:7:7:7::1 dev net0 weight 5· The reason why you got proto bgp is that when dumping the route, the kernel use the configs from user space(actually the type and protocol are merged to the first one and ignored at latest). > > and afterwards `ip route show` will give: > > > local a:b:c:f::/64 table 10222 proto kernel scope global metric 20 pref > medium > > nexthop via 7:7:7:7::1 dev net0 weight 5· > > nexthop via 7:7:7:7::1 dev net1 weight 1· > So when really dump the route, The type and protocol are used the first hops info. > > This is a severe problem, because > > 1) adding a "unicast" route will be merged with a "local" route, which seems > very wrong. > > 2) the netlink events are not correct. NetworkManager maintains a cache of > the routes. It requires that the netlink events allow to find out what is > happening. The wrong netlink notification makes it impossible for > NetworkManager to understand what happened. Netlink events are already > insufficient with `ip route replace` (bug 1337860), so when NetworkManager > sees an RTM_NEWROUTE with NLM_F_REPLACE flags, it already requests already > new dump of all routes to resync -- which is expensive. But in this case, > NetworkManager cannot tell from the netlink notification that something odd > happend, which would require a re-sync. > When IPv6 checks the routes, it only checks the dest address and devices. The type and protocol fields are ignored. > ... > So when really dump the route, The type and protocol are used the first hops info. Yes, I know :) I claim that that is a bug and it shouldn't be like that. (In reply to Thomas Haller from comment #6) > > When IPv6 checks the routes, it only checks the dest address and devices. > The type and protocol fields are ignored. > > ... > > So when really dump the route, The type and protocol are used the first hops info. > > Yes, I know :) > > I claim that that is a bug and it shouldn't be like that. As upstream suggested, it's hard to do the change on legacy nexthop api and the new nexthop api should resolve this issue. e.g. # ip addr add 2001:db8:101::1/64 dev dummy1 # ip addr add 2001:db8:101::2/64 dev dummy2 # ip nexthop add id 1 via 2001:db8:101::10 dev dummy1 # ip nexthop add id 2 via 2001:db8:101::10 dev dummy2 # ip nexthop add id 101 group 1/2 # ip nexthop show id 1 via 2001:db8:101::10 dev dummy1 scope link id 2 via 2001:db8:101::10 dev dummy2 scope link id 101 group 1/2 # ip route add table 100 2001:db8:102::/64 nhid 1 # ip route prepend table 100 2001:db8:102::/64 nhid 1 RTNETLINK answers: File exists # ip route add table 100 2001:db8:102::/64 nhid 2 RTNETLINK answers: File exists # ip route prepend table 100 2001:db8:102::/64 nhid 2 # ip -6 route show table 100 <- not merged 2001:db8:102::/64 nhid 1 via 2001:db8:101::10 dev dummy1 metric 1024 pref medium 2001:db8:102::/64 nhid 2 via 2001:db8:101::10 dev dummy2 metric 1024 pref medium # ip route add table 100 local 2001:db8:103::/64 nhid 1 # ip route prepend table 100 unicast 2001:db8:103::/64 nhid 1 RTNETLINK answers: File exists # ip route prepend table 100 unicast 2001:db8:103::/64 nhid 2 # ip -6 route show table 100 <- local/unicast are separated local 2001:db8:103::/64 nhid 1 via 2001:db8:101::10 dev dummy1 metric 1024 pref medium 2001:db8:103::/64 nhid 2 via 2001:db8:101::10 dev dummy2 metric 1024 pref medium # ip route add table 100 2001:db8:104::/64 nhid 1 proto kernel # ip route prepend table 100 2001:db8:104::/64 nhid 1 proto bgp RTNETLINK answers: File exists # ip route prepend table 100 2001:db8:104::/64 nhid 2 proto bgp # ip -6 route show table 100 <- kernel/bgp are separated 2001:db8:104::/64 nhid 1 via 2001:db8:101::10 dev dummy1 proto kernel metric 1024 pref medium 2001:db8:104::/64 nhid 2 via 2001:db8:101::10 dev dummy2 proto bgp metric 1024 pref medium # ip -6 route add table 100 local 2001:db8:105::/64 nhid 101 proto kernel # ip -6 route show table 100 local 2001:db8:105::/64 nhid 101 proto kernel metric 1024 pref medium nexthop via 2001:db8:101::10 dev dummy1 weight 1 nexthop via 2001:db8:101::10 dev dummy2 weight 1 # ip -6 route prepend table 100 unicast 2001:db8:105::/64 nhid 101 proto bgp RTNETLINK answers: File exists > it's hard to do the change on legacy nexthop api and the new nexthop api should resolve this issue. e.g.
Is that relevant?
Even if NetworkManager uses nexthops and never configures such a thing itself.
Any user can type such commands in the terminal. Kernel will do the wrong thing (of mangling existing routes). Kernel will also send wrong netlink notifictions, confusing anybody who cares about those notifications (e.g. NetworkManager).
Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug. This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there. Due to differences in account names between systems, some fields were not replicated. Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information. To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer. You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like: "Bugzilla Bug" = 1234567 In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information. |
With `ip -6 route append|prepend`, kernel wrongly merges routes that have a next-hop set. Actually, the kernel behavior is rather confusing (maybe even inconsistent and nonsensical?). It's hard to point out the single wrong thing with it. Let's just look at the two following examples, maybe if we understand/fix those, the broader problem is fixed/understandable: Example 1: # ip -6 route append unicast 1:2:3::6 dev v proto kernel table 5 # ip -6 route append blackhole 1:2:3::6 dev v proto kernel table 5 # ip -6 route append local 1:2:3::6 dev v proto kernel table 5 RTNETLINK answers: File exists or the other way around: # ip -6 route append local 1:2:3::6 dev v proto kernel table 5 # ip -6 route append unicast 1:2:3::6 dev v proto kernel table 5 RTNETLINK answers: File exists So we can add/append two routes that only differ by their type ("unicast" vs. "blackhole"), but not if the type is "local". The question is "what is a route?" (bug 1337855). A route (or policy routing rule) in kernel doesn't have a small set of attributes that make up an identity of the route. Instead, there is a larger set of attributes, that determine what a route "is". As seen above, we can configure two routes that only differ by their type ("unicast" and "blackhole"). That suggest that the type is part of the identity of a route. Note that altought the type is part of the identity of a route, we cannot add a type "local" route together with a type "unicast" route. OK, that may not yet invalidate the idea that the route type is part of the identity, it just means that despite a "local" route and a "unicast" route being different routes (all other attributes being equal), they cannot be configured at the same time. Kernel has various such inter-dependencies for other cases (e.g. when there is a next hop, there must be another route so that the gateway is reachable on-link). But see the attached reproducer... it behaves similar on Fedora 37 (6.0.18-300.fc37.x86_64) and rhel-9.2 (5.14.0-234.el9.x86_64). The only difference is that the `ip monitor` on rhel-9.2 doesn't output anything. Dunno why. Another bug? I'll attach the output of the script from Fedora 37. That script falls out form a unit test of NetworkManager, where NetworkManager configures randomly some routes and eventually it hits the problem. In this case, the problem is if routes have a next-hop. IPv6 routes with a next-hop are automatically merged by kernel. So in the script we see first that we have a local route: > local a:b:c:f::/64 via 7:7:7:7::1 dev net0 table 10222 proto kernel scope global metric 20 pref medium then we prepend a unicast route: # ip route prepend a:b:c:f::/64 proto bgp metric 20 table 10222 nexthop via 7:7:7:7::1 dev net1 we get a netlink notification for a unicast route (on Fedora only): > a:b:c:f::/64 table 10222 proto bgp metric 20 pref medium > nexthop via 7:7:7:7::1 dev net1 weight 1· > nexthop via 7:7:7:7::1 dev net0 weight 5· and afterwards `ip route show` will give: > local a:b:c:f::/64 table 10222 proto kernel scope global metric 20 pref medium > nexthop via 7:7:7:7::1 dev net0 weight 5· > nexthop via 7:7:7:7::1 dev net1 weight 1· So the merging seems really wrong. The netlink notification does also not match to what is happening (or doesn't come on rhel-9.2?). I think the problem is related to how "type local" routes are treated in the example above. The solution is possible to ensure that route with type "local" don't conflict/overlap/merge with "unicast" routes. The unit test in NetworkManager can probably uncover other scenarios. But let's address the attached reproducer first, and see afterwards whether other issues can be uncovered beyond that. Please reach out how to run NetworkManager's unit tests (it is currently patched to not cover the problematic case). This is a severe problem, because 1) adding a "unicast" route will be merged with a "local" route, which seems very wrong. 2) the netlink events are not correct. NetworkManager maintains a cache of the routes. It requires that the netlink events allow to find out what is happening. The wrong netlink notification makes it impossible for NetworkManager to understand what happened. Netlink events are already insufficient with `ip route replace` (bug 1337860), so when NetworkManager sees an RTM_NEWROUTE with NLM_F_REPLACE flags, it already requests already new dump of all routes to resync -- which is expensive. But in this case, NetworkManager cannot tell from the netlink notification that something odd happend, which would require a re-sync. PS: I still have hope that one day bug 1337855 and bug 1337860 could be resolved.