Bug 2091885
| Summary: | ipv6 default entry from RA stays after a change in the link-local address of the router | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 8 | Reporter: | Konstantinos <kkarampo> | ||||
| Component: | NetworkManager | Assignee: | Beniamino Galvani <bgalvani> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Matej Berezny <mberezny> | ||||
| Severity: | unspecified | Docs Contact: | |||||
| Priority: | unspecified | ||||||
| Version: | 8.4 | CC: | bgalvani, cback, dcritch, djuran, jgato, lrintel, mberezny, mcambria, nm-team, rkhan, sfaye, sukulkar, thaller, till, vbenes | ||||
| Target Milestone: | rc | Keywords: | Triaged, ZStream | ||||
| Target Release: | --- | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | NetworkManager-1.39.7-2.el9 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | |||||||
| : | 2096175 2097623 2097624 (view as bug list) | Environment: | |||||
| Last Closed: | 2022-11-08 10:10:38 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 2096175, 2097623, 2097624 | ||||||
| Attachments: |
|
||||||
I would expect that on a graceful stop of the router, an RA will be sent out with lifetime 0 so the route to be removed instantly. Or in case of no graceful shutdown the route entry to expire (Router life time 3600s in the RA). Hi, the default route has an expiration that is tracked internally to NM. Initially it is: <debug> [1653986718.9211] ndisc[0x55b5fe757cf0,"br-ex"]: gateway fe80::5054:ff:fe3d:d3f4 pref medium exp 3600.000 After the gateway changes, it becomes: <debug> [1653986796.3509] ndisc[0x55b5fe757cf0,"br-ex"]: gateway fe80::5054:ff:fe3d:d3f4 pref medium exp 3580.010 <debug> [1653986796.3510] ndisc[0x55b5fe757cf0,"br-ex"]: gateway fe80::1 pref medium exp 3600.000 ... <debug> [1653987214.5928] ndisc[0x55b5fe757cf0,"br-ex"]: gateway fe80::5054:ff:fe3d:d3f4 pref medium exp 3161.769 <debug> [1653987214.5928] ndisc[0x55b5fe757cf0,"br-ex"]: gateway fe80::1 pref medium exp 3600.000 The first route will be removed eventually when the gateway information from RA expires. (In reply to Konstantinos from comment #0) > In the node we have > ``` > [root@openshift-master-0 ~]# sysctl -a|grep "net.ipv6.conf.br-ex.accept_ra " > net.ipv6.conf.br-ex.accept_ra = 0 > ``` > Is that expected? I would have guessed that we need value `2` Yes, NM manages IPv6 in userspace and to do so disables some kernel sysctls. > I would expect that on a graceful stop of the router, an RA will be sent out with lifetime 0 so the route to be removed instantly
From the radvd.conf man page it seems that you need to explicitly set "AdvDefaultLifetime 0" to announce a zero lifetime for the gateway.
Thanks for the replies! > The first route will be removed eventually when the gateway information from RA expires. This is not happening AFAICT, is there a way to see the remaining expiring time per route entry(probably yes from logs but any other way)? Here are the logs after running around 2hours ``` May 31 08:46:16 openshift-master-0.hub-virtual.lab NetworkManager[3245]: <debug> [1653986776.3611] ndisc[0x55b5fe757cf0,"br-ex"]: gateway fe80::5054:ff:fe3d:d3f4 pref medium exp 3600.000 May 31 09:46:07 openshift-master-0.hub-virtual.lab NetworkManager[3245]: <debug> [1653990367.2916] ndisc[0x55b5fe757cf0,"br-ex"]: gateway fe80::5054:ff:fe3d:d3f4 pref medium exp 9.070 [root@openshift-master-0 ~]# date Tue May 31 10:56:31 UTC 2022 [root@openshift-master-0 ~]# ip -6 route|grep -A2 "default proto ra" default proto ra metric 49 pref medium nexthop via fe80::5054:ff:fe3d:d3f4 dev br-ex weight 1 nexthop via fe80::1 dev br-ex weight 1 [root@openshift-master-0 ~]# journalctl -b _SYSTEMD_UNIT=NetworkManager.service | grep "fe80::5054:ff:fe3d:d3f4" |tail -4 May 31 09:46:07 openshift-master-0.hub-virtual.lab NetworkManager[3245]: <debug> [1653990367.2916] ndisc[0x55b5fe757cf0,"br-ex"]: gateway fe80::5054:ff:fe3d:d3f4 pref medium exp 9.070 May 31 09:46:07 openshift-master-0.hub-virtual.lab NetworkManager[3245]: <debug> [1653990367.2923] platform: (br-ex) route: append IPv6 route: type unicast ::/0 via fe80::5054:ff:fe3d:d3f4 dev 11 metric 49 mss 0 rt-src ndisc May 31 09:46:07 openshift-master-0.hub-virtual.lab NetworkManager[3245]: <debug> [1653990367.2923] platform-linux: do-add-ip6-route[type unicast ::/0 via fe80::5054:ff:fe3d:d3f4 dev 11 metric 49 mss 0 rt-src rt-ra]: failure 17 (File exists) May 31 09:46:07 openshift-master-0.hub-virtual.lab NetworkManager[3245]: <debug> [1653990367.2923] platform: (br-ex) route-sync: adding route type unicast ::/0 via fe80::5054:ff:fe3d:d3f4 dev 11 metric 49 mss 0 rt-src ndisc failed with EEXIST, however we cannot find such a route ``` > From the radvd.conf man page it seems that you need to explicitly set "AdvDefaultLifetime 0" to announce a zero lifetime for the gateway. Ack I can check that also, nevertheless radvd is my setup and I cannot tell what happens with real routers. Thx! btw when I do `systemctl stop radvd` then by default I observe an RA
```
interface hub-baremetal
{
AdvSendAdvert on;
# Note: {Min,Max}RtrAdvInterval cannot be obtained with radvdump
AdvManagedFlag on;
AdvOtherConfigFlag off;
AdvReachableTime 0;
AdvRetransTimer 0;
AdvCurHopLimit 64;
AdvDefaultLifetime 0;
AdvHomeAgentFlag off;
AdvDefaultPreference medium;
AdvSourceLLAddress on;
prefix 2620:52:0:1305::/64
{
AdvValidLifetime 86400;
AdvPreferredLifetime 14400;
AdvOnLink on;
AdvAutonomous on;
AdvRouterAddr on;
}; # End of prefix definition
route ::/0
{
AdvRoutePreference low;
AdvRouteLifetime 0;
}; # End of route definition
};
```
and probably because of that
```
RemoveRoute on|off
Upon shutdown, announce this route with a zero second lifetime. This should cause the route to be immediately removed from the receiving end-nodes' route table.
Default: on
```
(In reply to Konstantinos from comment #6) > Thanks for the replies! > > > The first route will be removed eventually when the gateway information from RA expires. > > This is not happening AFAICT Can you attach full logs? > is there a way to see the remaining expiring > time per route entry(probably yes from logs but any other way)? At the moment no, it's only in logs. I think NM should set the lifetime also in kernel so that it can be seen with "ip route". (In reply to Konstantinos from comment #7) > btw when I do `systemctl stop radvd` then by default I observe an RA > ... > and probably because of that > ``` > RemoveRoute on|off Right, good catch. I have attached logs after almost 5hours.
> I think NM should set the lifetime also in kernel so that it can be seen with "ip route"
I see that this is not the case for me, (I try ip -6 -d route), is it a bug?
> > The first route will be removed eventually when the gateway information from RA expires. > > This is not happening AFAICT It's a bug in NetworkManager caused by the presence of a multipath IPv6 route. When there are two default routes, the kernel merges them as: default proto ra metric 101 pref medium nexthop via fe80::38e1:1bff:fe6a:e266 dev eth0 weight 1 nexthop via fe80::1 dev eth0 weight 1 Before commit [1], NM doesn't parse multipath routes and so it might get a partial vision of what is configured in kernel. Specifically, it is aware only of one of the next-hops; then when one of the two routes expires and must be removed, NM could decide that the route is already missing in kernel and no action is necessary. [1] https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/dac12a8d6178a6d82fc0913ad8825c8556380ba8 (In reply to Konstantinos from comment #11) > > I think NM should set the lifetime also in kernel so that it can be seen with "ip route" > I see that this is not the case for me, (I try ip -6 -d route), is it a bug? I meant, ideally it should do that in the future. It doesn't at the moment. Created attachment 1885802 [details]
Reproducer script
Moving to MODIFIED as this is already fixed in RHEL 8.7. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (NetworkManager bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:7680 |
Description of problem: In an ipv6 only deployment of OCP 4.10: The router sends RAs with link-local ip fe80::5054:ff:fe3d:d3f4/64. For a reason (new device, VRRP change) the link-local address changes. The OCP nodes have wrong default routing. Version-Release number of selected component (if applicable): ```` oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.10.5 True True 18d [root@openshift-master-0 ~]# nmcli --version nmcli tool, version 1.30.0-13.el8_4 [root@openshift-master-0 ~]# uname -r 4.18.0-305.40.2.el8_4.x86_64 ``` How reproducible: Steps to Reproduce: 1. Have a router sending RA (e.g. radvd) 2. Change the link-local address ``` sudo ip add add fe80::1/64 dev hub-baremetal sudo ip add del fe80::5054:ff:fe3d:d3f4/64 dev hub-baremetal sudo systemctl restart radvd ``` 3. Inspect the routing table in the nodes `ip -6 -d route` for the default. Actual results: ``` [root@openshift-master-0 ~]# ndptool monitor -t ra [50/382] NDP payload len 80, from addr: fe80::5054:ff:fe3d:d3f4, iface: br-ex Type: RA Hop limit: 64 Managed address configuration: yes Other configuration: no Default router preference: medium Router lifetime: 3600s Reachable time: unspecified Retransmit time: unspecified Source linkaddr: 52:54:00:3d:d3:f4 Prefix: 2620:52:0:1305::/64, valid_time: 86400s, preferred_time: 14400s, on_link: yes, autonomous_addr_conf: yes, router_addr: yes Route: ::/0, lifetime: 9000s, preference: low [root@openshift-master-0 ~]# ip -6 -d route ... unicast default via fe80::5054:ff:fe3d:d3f4 dev br-ex proto ra scope global metric 49 pref medium ``` After ``` [root@openshift-master-0 ~]# ndptool monitor -t ra NDP payload len 80, from addr: fe80::1, iface: br-ex <---- link-local change Type: RA Hop limit: 64 Managed address configuration: yes Other configuration: no Default router preference: medium Router lifetime: 3600s Reachable time: unspecified Retransmit time: unspecified Source linkaddr: 52:54:00:3d:d3:f4 Prefix: 2620:52:0:1305::/64, valid_time: 86400s, preferred_time: 14400s, on_link: yes, autonomous_addr_conf: yes, router_addr: yes [root@openshift-master-0 ~]# ip -6 -d route unicast default proto ra scope global metric 49 pref medium nexthop via fe80::5054:ff:fe3d:d3f4 dev br-ex weight 1 <-- this is dead entry nexthop via fe80::1 dev br-ex weight 1 ``` Expected results: This entry not to exist ``` nexthop via fe80::5054:ff:fe3d:d3f4 dev br-ex weight 1 ``` and to observe expire information when doing an `ip -6 route` e.g ``` default via fe80::1a8b:9dff:fed4:822 dev enp0s31f6 proto ra metric 1024 expires 1798sec mtu 1500 pref medium ``` Additional info: In the node we have ``` [root@openshift-master-0 ~]# sysctl -a|grep "net.ipv6.conf.br-ex.accept_ra " net.ipv6.conf.br-ex.accept_ra = 0 ``` Is that expected? I would have guessed that we need value `2` Why we do not see the expires information?