I would expect that on a graceful stop of the router, an RA will be sent out with lifetime 0 so the route to be removed instantly. Or in case of no graceful shutdown the route entry to expire (Router life time 3600s in the RA).

Hi,

the default route has an expiration that is tracked internally to NM.

Initially it is:

 <debug> [1653986718.9211] ndisc[0x55b5fe757cf0,"br-ex"]:   gateway fe80::5054:ff:fe3d:d3f4 pref medium exp 3600.000

After the gateway changes, it becomes:

 <debug> [1653986796.3509] ndisc[0x55b5fe757cf0,"br-ex"]:   gateway fe80::5054:ff:fe3d:d3f4 pref medium exp 3580.010
 <debug> [1653986796.3510] ndisc[0x55b5fe757cf0,"br-ex"]:   gateway fe80::1 pref medium exp 3600.000

 ...

 <debug> [1653987214.5928] ndisc[0x55b5fe757cf0,"br-ex"]:   gateway fe80::5054:ff:fe3d:d3f4 pref medium exp 3161.769
 <debug> [1653987214.5928] ndisc[0x55b5fe757cf0,"br-ex"]:   gateway fe80::1 pref medium exp 3600.000

The first route will be removed eventually when the gateway information from RA expires.

(In reply to Konstantinos from comment #0)
> In the node we have
> ```
> [root@openshift-master-0 ~]# sysctl -a|grep "net.ipv6.conf.br-ex.accept_ra "
> net.ipv6.conf.br-ex.accept_ra = 0
> ```
> Is that expected? I would have guessed that we need value `2`

Yes, NM manages IPv6 in userspace and to do so disables some kernel sysctls.

> I would expect that on a graceful stop of the router, an RA will be sent out with lifetime 0 so the route to be removed instantly

From the radvd.conf man page it seems that you need to explicitly set "AdvDefaultLifetime 0" to announce a zero lifetime for the gateway.

Thanks for the replies!

> The first route will be removed eventually when the gateway information from RA expires.

This is not happening AFAICT, is there a way to see the remaining expiring time per route entry(probably yes from logs but any other way)?

Here are the logs after running around 2hours
```
May 31 08:46:16 openshift-master-0.hub-virtual.lab NetworkManager[3245]: <debug> [1653986776.3611] ndisc[0x55b5fe757cf0,"br-ex"]:   gateway fe80::5054:ff:fe3d:d3f4 pref medium exp 3600.000
May 31 09:46:07 openshift-master-0.hub-virtual.lab NetworkManager[3245]: <debug> [1653990367.2916] ndisc[0x55b5fe757cf0,"br-ex"]:   gateway fe80::5054:ff:fe3d:d3f4 pref medium exp 9.070
[root@openshift-master-0 ~]# date
Tue May 31 10:56:31 UTC 2022
[root@openshift-master-0 ~]# ip -6 route|grep -A2 "default proto ra"
default proto ra metric 49 pref medium
        nexthop via fe80::5054:ff:fe3d:d3f4 dev br-ex weight 1
        nexthop via fe80::1 dev br-ex weight 1


[root@openshift-master-0 ~]# journalctl -b _SYSTEMD_UNIT=NetworkManager.service | grep "fe80::5054:ff:fe3d:d3f4" |tail -4
May 31 09:46:07 openshift-master-0.hub-virtual.lab NetworkManager[3245]: <debug> [1653990367.2916] ndisc[0x55b5fe757cf0,"br-ex"]:   gateway fe80::5054:ff:fe3d:d3f4 pref medium exp 9.070
May 31 09:46:07 openshift-master-0.hub-virtual.lab NetworkManager[3245]: <debug> [1653990367.2923] platform: (br-ex) route: append     IPv6 route: type unicast ::/0 via fe80::5054:ff:fe3d:d3f4 dev 11 metric 49 mss 0 rt-src ndisc
May 31 09:46:07 openshift-master-0.hub-virtual.lab NetworkManager[3245]: <debug> [1653990367.2923] platform-linux: do-add-ip6-route[type unicast ::/0 via fe80::5054:ff:fe3d:d3f4 dev 11 metric 49 mss 0 rt-src rt-ra]: failure 17 (File exists)
May 31 09:46:07 openshift-master-0.hub-virtual.lab NetworkManager[3245]: <debug> [1653990367.2923] platform: (br-ex) route-sync: adding route type unicast ::/0 via fe80::5054:ff:fe3d:d3f4 dev 11 metric 49 mss 0 rt-src ndisc failed with EEXIST, however we cannot find such a route
```


> From the radvd.conf man page it seems that you need to explicitly set "AdvDefaultLifetime 0" to announce a zero lifetime for the gateway.
Ack I can check that also, nevertheless radvd is my setup and I cannot tell what happens with real routers.

Thx!

btw when I do `systemctl stop radvd` then by default I observe an RA
```
interface hub-baremetal
{
        AdvSendAdvert on;
        # Note: {Min,Max}RtrAdvInterval cannot be obtained with radvdump
        AdvManagedFlag on;
        AdvOtherConfigFlag off;
        AdvReachableTime 0;
        AdvRetransTimer 0;
        AdvCurHopLimit 64;
        AdvDefaultLifetime 0;
        AdvHomeAgentFlag off;
        AdvDefaultPreference medium;
        AdvSourceLLAddress on;

        prefix 2620:52:0:1305::/64
        {
                AdvValidLifetime 86400;
                AdvPreferredLifetime 14400;
                AdvOnLink on;
                AdvAutonomous on;
                AdvRouterAddr on;
        }; # End of prefix definition


        route ::/0
        {
                AdvRoutePreference low;
                AdvRouteLifetime 0;
        }; # End of route definition

}; 

```

and probably because  of that 
```
RemoveRoute on|off
Upon shutdown, announce this route with a zero second lifetime. This should cause the route to be immediately removed from the receiving end-nodes' route table.

Default: on
```

(In reply to Konstantinos from comment #6)
> Thanks for the replies!
> 
> > The first route will be removed eventually when the gateway information from RA expires.
> 
> This is not happening AFAICT

Can you attach full logs?

> is there a way to see the remaining expiring
> time per route entry(probably yes from logs but any other way)?

At the moment no, it's only in logs. I think NM should set the lifetime also in kernel so that it can be seen with "ip route".

(In reply to Konstantinos from comment #7)
> btw when I do `systemctl stop radvd` then by default I observe an RA
> ...
> and probably because  of that 
> ```
> RemoveRoute on|off

Right, good catch.

I have attached logs after almost 5hours.

> I think NM should set the lifetime also in kernel so that it can be seen with "ip route"
I see that this is not the case for me, (I try ip -6 -d route), is it a bug?

> > The first route will be removed eventually when the gateway information from RA expires.
>
> This is not happening AFAICT

It's a bug in NetworkManager caused by the presence of a multipath
IPv6 route. When there are two default routes, the kernel merges them
as:

 default proto ra metric 101 pref medium
	 nexthop via fe80::38e1:1bff:fe6a:e266 dev eth0 weight 1
	 nexthop via fe80::1 dev eth0 weight 1

Before commit [1], NM doesn't parse multipath routes and so it might
get a partial vision of what is configured in kernel. Specifically, it
is aware only of one of the next-hops; then when one of the two routes
expires and must be removed, NM could decide that the route is already
missing in kernel and no action is necessary.

[1] https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/commit/dac12a8d6178a6d82fc0913ad8825c8556380ba8

(In reply to Konstantinos from comment #11)
> > I think NM should set the lifetime also in kernel so that it can be seen with "ip route"
> I see that this is not the case for me, (I try ip -6 -d route), is it a bug?

I meant, ideally it should do that in the future. It doesn't at the moment.

Created attachment 1885802 [details]
Reproducer script

Moving to MODIFIED as this is already fixed in RHEL 8.7.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (NetworkManager bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:7680