To demonstrate accept_ra fixing the problem:

1. Ensure NetworkManager 1.20.0-5 is installed
2. Configure ipv6.token on an interface (or IPV6_TOKEN in ifcfg file) to say "::123"
3. Apply changes as necessary ("nmcli c reload; nmcli c up ens3" or reboot)
4. Check "ip token" output and /var/log/messages
5. Enable accept_ra sysctl (e.g. "echo 1 > /proc/sys/net/ipv6/conf/ens3/accept_ra")
6. Check "ip token" output and /var/log/messages again


And a demonstration with ip commands:

[root@host ~]# echo 0 > /proc/sys/net/ipv6/conf/ens3/accept_ra
[root@host ~]# ip token set ::123 dev ens3
RTNETLINK answers: Invalid argument
[root@host ~]# echo 1 > /proc/sys/net/ipv6/conf/ens3/accept_ra
[root@host ~]# ip token set ::155:198:51:3 dev ens3
[root@host ~]#

this seems to be due to a kernel misfeature, which refuses to accept the token, if accept_ra is disabled.

Interestingly, if you have


# echo 1 >  /proc/sys/net/ipv6/conf/w/accept_ra
# ip token set ::123 dev w
# ip token
token ::123 dev w
# echo 0 >  /proc/sys/net/ipv6/conf/w/accept_ra
# ip token
token ::123 dev w
# ip token set ::124 dev w
RTNETLINK answers: Invalid argument


Meaning, if accept_ra is disabled, you cannot set the token, but kernel would still happily report it as set.

I guess, one workaround could be to quickly toggle accept_ra while link_set_token() in src/platform/nm-linux-platform.c


Seems the worst that happens is the spamming of the logfile with error messages. The failure to actually set the token doesn't seem too serious, because NetworkManager generates the interface identifiers in user-space and will honor the token there.

Hi,

Thanks for looking at this and raising the bug against the kernel component. I've encountered another consequence of this issue. When NetworkManager is used in conjunction with the legacy network-scripts package and IPV6_TOKEN option, on reboot the machine acquires both an EUI-64 address, as well as the desired token address. This is a problem, as the IP address of outbound traffic is then ambiguous. Downgrading to NetworkManager 1.20.0-3 and rebooting results in the expected behaviour of only having a token address. I suspect the interaction between network-scripts and NetworkManager requires the token to be present in the kernel.

I realise network-scripts is deprecated, but I am not aware of any other supported way to configure additional addresses on the loopback interface, which we use on machines with VIPs outside of the local subnet (e.g. when doing BGP RHI or putting a static route on the upstream router). Additionally, it doesn't seem possible to use NetworkManager to configure a default route within a different routing table. I.e. the following setup is not possible with NetworkManager alone:

# ifcfg-eth0
IPADDR="192.168.0.2"
NETMASK="255.255.255.0"
GATEWAY="192.168.0.1"

# ifcfg-eth1
IPADDR="10.0.0.2"
NETMASK="255.255.255.0"

# route-eth1
default via 10.0.0.1 table 100

# rule-eth1
from 10.0.0.2/32 table 100

Cheers,

David.

Upstream merge request to fix setting kernel token:

https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/519

(In reply to David Stockdale from comment #3)
> Thanks for looking at this and raising the bug against the kernel component.
> I've encountered another consequence of this issue. When NetworkManager is
> used in conjunction with the legacy network-scripts package and IPV6_TOKEN
> option, on reboot the machine acquires both an EUI-64 address, as well as
> the desired token address.

I couldn't reproduce this on a RHEL 8.2 machine. Can you please set level=TRACE in the [logging] section of /etc/NetworkManager/NetworkManager.conf, set RateLimitIntervalSec=0 in /etc/systemd/journald.conf to disable journald rate limiting, reboot and attach the output of 'journalctl -b'?

ipv6_token test added:
https://gitlab.freedesktop.org/NetworkManager/NetworkManager-ci/-/merge_requests/592

Created attachment 1700195 [details]
EUI-64 address persists when using IPV6_TOKEN and legacy network-scripts

Hi,

Sorry for taking so long to reply. I just recreated the issue whereby with legacy network-scripts package installed and IPV6_TOKEN configured, I end up with an undesired EUI-64 address too. Requested log attached.

Output of "ip addr" after boot is:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:42:d4:89 brd ff:ff:ff:ff:ff:ff
    inet 155.198.30.47/23 brd 155.198.31.255 scope global dynamic noprefixroute ens3
       valid_lft 258839sec preferred_lft 258839sec
    inet6 2a0c:5bc0:80:117:5054:ff:fe42:d489/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 544sec preferred_lft 244sec
    inet6 2001:630:12:602:5054:ff:fe42:d489/64 scope global deprecated dynamic mngtmpaddr noprefixroute 
       valid_lft 542sec preferred_lft 0sec
    inet6 2a0c:5bc0:80:117::47/64 scope global dynamic noprefixroute 
       valid_lft 900sec preferred_lft 600sec
    inet6 2001:630:12:602::47/64 scope global deprecated dynamic noprefixroute 
       valid_lft 900sec preferred_lft 0sec
    inet6 fe80::5054:ff:fe42:d489/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

I'm afraid that's a bit confusing, as there is an old prefix present (2001:630:12:602::/64) that is soon to be retired. Last time I saw this, it was on a network with only one prefix advertised in the RAs.

Incidentally, with network-scripts installed (and chkconfig on), the kernel gets updated with the IPv6 token - "ip token" looks correct and there are no errors in syslog.

Package versions:

NetworkManager-1.22.8-4.el8.x86_64
network-scripts-10.00.6-1.el8_2.1.x86_64

Unfortunately on this newer version of NetworkManager, I am hitting https://bugzilla.redhat.com/show_bug.cgi?id=1841937 (had two machines drop off the network since updating them). I think I'll have to pin our servers to 1.20.0-3 until there is a package that fixes both these bugs. Do you know when that might be? I see 1.25.2-1 should fix both issues, but I think 1.25 is a dev release? Will these fixes be backported?

Cheers,

David.

(In reply to David Stockdale from comment #9)
> Created attachment 1700195 [details]
> EUI-64 address persists when using IPV6_TOKEN and legacy network-scripts

It's unclear to me what you did here. What is this supposed to show?

The log shows that you use the deprecated "network-scripts" (initscripts). But as you also have NetworkManager running (and no "NM_CONTROLLED=no" in the ifcfg-file), ifup calls to NetworkManager. As you also see in the attached log, that NetworkManager handles the interface. In this case, this has very little to do with initscripts (and the IPV6_TOKEN variable).

Only surprising is that NetworkManager is setting token ::47, under seemingly the same circumstances as comment 0, but in this case it doesn't fail.


> I'm afraid that's a bit confusing, as there is an old prefix present (2001:630:12:602::/64) that is soon to be retired.

As you see in the logs, this is because the router advertises this prefix. Check the router advertisements with tcpdump or `ndptool monitor -i ens3`.


> Unfortunately on this newer version of NetworkManager, I am hitting https://bugzilla.redhat.com/show_bug.cgi?id=1841937 (had two machines drop off the network since updating them). I think I'll have to pin our servers to 1.20.0-3 until there is a package that fixes both these bugs. Do you know when that might be? I see 1.25.2-1 should fix both issues, but I think 1.25 is a dev release? Will these fixes be backported?

NetworkManager tries to set the token in kernel, but due to this bug, that might fail.
In fact, there are little negative effects (has it?), aside the <warn> message in the log. The token even works, because NetworkManager generates the full IPv6 address in user-space, it doesn't matter which token you set in kernel.

That is why this is not planned to be backported to 8.2. It will be fixed in 8.3.


On the other hand, bug 1841937 will be fixed by the next 8.2 z-stream update. Please ask our support to know when (and how) you get to the update. Or, as another workaround, you could configure  'dhcp=dhclient' or 'dhcp=systemd' in the [main] section of /etc/NetworkManager/NetworkManager.conf. Then restart NM with 'systemctl restart NetworkManager'. That uses a different DHCP implementation which hasn't this issue.

(In reply to Thomas Haller from comment #10)
> (In reply to David Stockdale from comment #9)
> > Created attachment 1700195 [details]
> > EUI-64 address persists when using IPV6_TOKEN and legacy network-scripts
> 
> It's unclear to me what you did here. What is this supposed to show?

I did as you asked in your previous comment:

"Can you please set level=TRACE in the [logging] section of /etc/NetworkManager/NetworkManager.conf, set RateLimitIntervalSec=0 in /etc/systemd/journald.conf to disable journald rate limiting, reboot and attach the output of 'journalctl -b'?"

> The log shows that you use the deprecated "network-scripts" (initscripts).
> But as you also have NetworkManager running (and no "NM_CONTROLLED=no" in
> the ifcfg-file), ifup calls to NetworkManager. As you also see in the
> attached log, that NetworkManager handles the interface. In this case, this
> has very little to do with initscripts (and the IPV6_TOKEN variable).

As I mentioned above, the reason for using network-scripts is to be able to set VIPs on the loopback interface, which I am not aware of any way to do with NetworkManager. This is why ens3 is left under NetworkManager control, as it's only lo that we're interested in. In fact, it is desirable to use NetworkManager for ens3, as I don't think network-scripts implements IPV6_TOKEN.

The other use case we have for network-scripts, is to be able to have additional routing tables with default routes, though in that case we would have to disable NetworkManager.

> Only surprising is that NetworkManager is setting token ::47, under
> seemingly the same circumstances as comment 0, but in this case it doesn't
> fail.

Yep, curious.

> > I'm afraid that's a bit confusing, as there is an old prefix present (2001:630:12:602::/64) that is soon to be retired.
> 
> As you see in the logs, this is because the router advertises this prefix.
> Check the router advertisements with tcpdump or `ndptool monitor -i ens3`.

I know why the extra addresses are being configured (and they are intentionally deprecated by setting a preferred lifetime of zero on the router, until we have finished renumbering all hosts). I was just mentioning that I don't think the multiple prefixes was the cause of the problem.

> > Unfortunately on this newer version of NetworkManager, I am hitting https://bugzilla.redhat.com/show_bug.cgi?id=1841937 (had two machines drop off the network since updating them). I think I'll have to pin our servers to 1.20.0-3 until there is a package that fixes both these bugs. Do you know when that might be? I see 1.25.2-1 should fix both issues, but I think 1.25 is a dev release? Will these fixes be backported?
> 
> NetworkManager tries to set the token in kernel, but due to this bug, that
> might fail.
> In fact, there are little negative effects (has it?), aside the <warn>
> message in the log. The token even works, because NetworkManager generates
> the full IPv6 address in user-space, it doesn't matter which token you set
> in kernel.
> 
> That is why this is not planned to be backported to 8.2. It will be fixed in
> 8.3.

In our case, when using network-scripts, the ip token issue seems to be causing the unwanted EUI-64 addresses (doesn't happen with 1.20.0-3). This is a problem for us, as we need the IPv6 address of outbound connections to be stable. I'll stick with the older version for the time being. Good to know that 8.3 should bring the fix.

> In our case, when using network-scripts, the ip token issue seems to be causing the unwanted EUI-64 addresses (doesn't happen with 1.20.0-3). This is a problem for us, as we need the IPv6 address of outbound connections to be stable. I'll stick with the older version for the time being. Good to know that 8.3 should bring the fix.

you mean addresses 2a0c:5bc0:80:117:5054:ff:fe42:d489/64 and 2001:630:12:602:5054:ff:fe42:d489/64?

It's not clear who adds them, if you look at the log, the just appear (seemingly not added by NetworkManager). 

Also, neither this bug nor 1841937 is not about that address.

Beniamino, what do you think?

Yep, those addresses. My guess is that they are being configured by the kernel, during a window when the interface is up, accept_ra=1, autoconf=1 and the ip token has not yet been set. Also, if accept_ra=1 at some point, that could explain how NetworkManager manages to set the ip token in the kernel, which it normally fails to do. This state only seems to arise when network-scripts are in play.

Experimenting a bit, if I start with the interface having EUI-64 addresses only:

2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:42:d4:89 brd ff:ff:ff:ff:ff:ff
    inet 155.198.30.47/23 brd 155.198.31.255 scope global dynamic noprefixroute ens3
       valid_lft 259158sec preferred_lft 259158sec
    inet6 2a0c:5bc0:80:117:5054:ff:fe42:d489/64 scope global dynamic noprefixroute 
       valid_lft 898sec preferred_lft 598sec
    inet6 2001:630:12:602:5054:ff:fe42:d489/64 scope global deprecated dynamic noprefixroute 
       valid_lft 898sec preferred_lft 0sec
    inet6 fe80::5054:ff:fe42:d489/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

Then set accept_ra=1 and add an ip token:

# sysctl net.ipv6.conf.ens3.accept_ra=1
# ip token set ::47 dev ens3

The EUI-64 addresses get stripped off when the tokenised addresses are configured:

2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:42:d4:89 brd ff:ff:ff:ff:ff:ff
    inet 155.198.30.47/23 brd 155.198.31.255 scope global dynamic noprefixroute ens3
       valid_lft 259088sec preferred_lft 259088sec
    inet6 2a0c:5bc0:80:117::47/64 scope global dynamic noprefixroute 
       valid_lft 899sec preferred_lft 599sec
    inet6 2001:630:12:602::47/64 scope global deprecated dynamic noprefixroute 
       valid_lft 899sec preferred_lft 0sec
    inet6 fe80::5054:ff:fe42:d489/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

I suspect the above is what happens, as desired, with NetworkManager 1.20.0-3 and network-scripts installed. I speculate that with the more recent versions of NetworkManager manipulating accept_ra and at times failing to set the ip token in the kernel, there is some undesirable interaction with the kernel address management (only occurring when network-scripts present) that causes the kernel to configure EUI-64 addresses and not strip them off when the tokenised addresses are configured.

Hi,

it seems the 2001:630:12:602:5054:ff:fe42:d489/64 address and the default route are added by the kernel; this shouldn't happen because at that point NM has set accept_ra=0. So perhaps there is something changing accept_ra back to 1, perhaps network-scripts. 

However network-scripts should just call NM when NM is running, so it's unclear why this happens. Can you please add 'set -x' at the top of /etc/sysconfig/network-scripts/ifup (below '#!/usr/bin/bash'), reboot and get new logs?

> Also, if accept_ra=1 at some point, that could explain how NetworkManager manages to set the ip token in the kernel, which it normally fails to do.

Good point. Yes, something changes accept_ra back to enabled. NM doesn't want that, and it causes the additional address we see.

I agree with Beniamino. Can you provide the debug log from comment 14? Thanks.

Created attachment 1700646 [details]
network-scripts and NetworkManager with IPV6_TOKEN

Thanks for the "set +x" tip! I added it to the ifup script, but there was still no sign of accept_ra being changed, but having grepped the other scripts, I found it's also set by init.ipv6-global, which is called from the network init script. I added a "set +x" to that too and sure enough you see this happen:

Jul 10 19:52:20 nsgtest.net.ic.ac.uk NetworkManager[988]: <debug> [1594407140.9929] platform-linux: sysctl: reading '/proc/sys/net/ipv6/conf/ens3/accept_ra': '1'
Jul 10 19:52:20 nsgtest.net.ic.ac.uk NetworkManager[988]: <debug> [1594407140.9952] platform-linux: sysctl: setting '/proc/sys/net/ipv6/conf/ens3/accept_ra' to '0' (current value is '1')
Jul 10 19:52:21 nsgtest.net.ic.ac.uk NetworkManager[988]: <debug> [1594407141.0264] platform-linux: sysctl: setting '/proc/sys/net/ipv6/conf/ens3/accept_ra' to '0' (current value is identical)
Jul 10 19:52:21 nsgtest.net.ic.ac.uk NetworkManager[988]: <debug> [1594407141.0466] platform-linux: sysctl: setting '/proc/sys/net/ipv6/conf/ens3/accept_ra' to '0' (current value is identical)
Jul 10 19:52:23 nsgtest.net.ic.ac.uk network[1054]: + /sbin/sysctl -e -w net.ipv6.conf.all.accept_ra=1
Jul 10 19:52:23 nsgtest.net.ic.ac.uk network[1054]: + /sbin/sysctl -e -w net.ipv6.conf.default.accept_ra=1
Jul 10 19:52:23 nsgtest.net.ic.ac.uk network[1054]: + /sbin/sysctl -e -w net.ipv6.conf.ens3.accept_ra=1
Jul 10 19:52:23 nsgtest.net.ic.ac.uk network[1054]: + /sbin/sysctl -e -w net.ipv6.conf.lo.accept_ra=1
Jul 10 19:52:25 nsgtest.net.ic.ac.uk NetworkManager[988]: <trace> [1594407145.9205] platform-linux: event-notification: RTM_NEWADDR, flags 0, seq 0: 2001:630:12:602:5054:ff:fe42:d489/64 lft 900sec pref 0sec lifetime 7-7[0,900] dev 2 flags deprecated,mngtmpaddr,tentative src kernel
Jul 10 19:52:25 nsgtest.net.ic.ac.uk NetworkManager[988]: <debug> [1594407145.9205] platform: (ens3) signal: address 6   added: 2001:630:12:602:5054:ff:fe42:d489/64 lft 900sec pref 0sec lifetime 7-7[0,900] dev 2 flags deprecated,mngtmpaddr,tentative src kernel
Jul 10 19:52:25 nsgtest.net.ic.ac.uk NetworkManager[988]: <debug> [1594407145.9242] platform: (ens3) address: adding or updating IPv6 address: 2001:630:12:602:5054:ff:fe42:d489/64 lft 900sec pref 0sec lifetime 7-0[0,900] dev 2 flags deprecated,mngtmpaddr,noprefixroute,tentative src unknown
Jul 10 19:52:25 nsgtest.net.ic.ac.uk NetworkManager[988]: <debug> [1594407145.9242] platform-linux: do-add-ip6-address[2: 2001:630:12:602:5054:ff:fe42:d489]: success

Furthermore, looking at init.ipv6-global, the sysctl settings are conditional on IPV6_SET_SYSCTLS not being set to "no". Having set that thus in /etc/sysconfig/network all is well. Perhaps this has been encountered before. Log attached.

With that issue solved, this bug does indeed seem largely cosmetic as you say (and can be worked around in rsyslog). I'll take a look at the workaround for the DHCP bug, now this issue is no longer a blocker to upgrading NetworkManager. Many thanks.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (NetworkManager bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4499