Bug 1548237
| Summary: | network interface ipv6 address missed after down and up it while NetworkManger service is running | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Frank Liang <xiliang> |
| Component: | NetworkManager | Assignee: | Beniamino Galvani <bgalvani> |
| Status: | CLOSED ERRATA | QA Contact: | Desktop QE <desktop-qa-list> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 7.5 | CC: | atragler, bgalvani, cheshi, fgiudici, jmaxwell, leiwang, linl, lrintel, rkhan, sukulkar, thaller, vbenes, vkuznets, wshi, xiliang |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | x86_64 | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | NetworkManager-1.18.0-2.el7 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-08-06 13:16:20 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
| Bug Depends On: | |||
| Bug Blocks: | 1654714 | ||
If you set a interface down (ip link set down $NAME), then kernel will remove all IPv6 addresses.
NetworkManager configures all links with "addrgenmode none". This tells kernel not to automatically add IPv6 link local addresses. Link local addresses are instead added by NetworkManager. This is necessary because:
- NetworkManager always sets links IFF_UP, even if the link is logically
disconnected. It does so, to get carrier-changed events from kernel. But if
the link is logically disconnected, it must have no IP addresses.
- support private stable IPv6 link local addresses (RFC7217).
See addr-gen-mode in `man nm-settings`.
You can see that with
ip -d link show
If you don't use NetworkManager, the link commonly has "addgenmode eui64". In that case, setting a link down will remove IPv6 addresses but bringing it up again will prompt kernel to re-add the IPv6 link local address. If you re-up a link with "addrgenmode none", nobody is re-adding the IPv6 link local address.
Not even NetworkManager will re-add the IPv6 address. In general, it's probably a bad idea to mess with interfaces that are managed by NetworkManager. But if you do so, NetworkManager tries hard not to interfere with your external changes. In this case, you set the link down and kernel removed the IPv6 addresses. NetworkManager sees that somebody did something, but it won't revert the external changes (how would it know to do so?).
You might wanna call `nmcli device reapply "$NAME`, which basically tells NetworkManager to restore all previously configured IP addresses. Or go through a full down+up cycle with `nmcli connection up "$CON"`.
> Additional info:
> - Cannot reproduce it on Fedora27 and Ubuntu.
Which versions of NetworkManager specifically? That sounds like a bug. It also doesn't sound very plausible, because Fedora27 and RHEL-7.4 both ship a rather similar NetworkManager (1.8.x) -- and below you said, that RHEL-7.4 shows the issue.
> - Cannot reproduce it if only enable network.service
In that case, the link likely is "addrgenmode eui64". Note that this will only restore IPv6 link local addresses. All other IPv6 addresses are gone -- arguably, kernel (depending on sysctl configuration) might do autoconf again and receive the same prefixes again. But manually added addresses are lost either way.
> - Can reproduce this bug with RHEL7.4 GA release installed, so it is not a
> regression.
As said, unclear why Fedora 27 would behave differently then rhel-7.4.
> - Only can reproduce it if enable NetworkManager service
As explained, that's because NetworkManager sets "addrgenmode none".
> - Can reproduce it in bare metal system(tg3 nic driver), so it is not a xen
> related bug
> - Can reproduce it both using "ip" or "ifconfig" to down and up network
> interface.
Both is not surprising.
I would argue this works as expected. Only thing that surprises me is the claim that Fedora 27 and Ubuntu behave differently. That would need some investigation/clarification.
Can you explain why you are setting the interface down in the first place? What are you trying to achive? Thanks
Sidenote: often logfiles of NetworkManager with level=TRACE are helpful. See hints about logging at https://cgit.freedesktop.org/NetworkManager/NetworkManager/tree/contrib/fedora/rpm/NetworkManager.conf
(In reply to Thomas Haller from comment #2) Thanks for your detail explanation. So, the behavior under RHEL7.5 is expected as "addrgenmode none". > > Additional info: > > - Cannot reproduce it on Fedora27 and Ubuntu. > > Which versions of NetworkManager specifically? That sounds like a bug. It > also doesn't sound very plausible, because Fedora27 and RHEL-7.4 both ship a > rather similar NetworkManager (1.8.x) -- and below you said, that RHEL-7.4 > shows the issue. Below is the NetworkManager version in Fedora27, RHEL7.5 and RHEL8 nightly build. NetworkManager-1.8.2-3.fc27.2.x86_64 NetworkManager-1.10.2-13.el7.x86_64 NetworkManager-1.10.4-1.el8+5.x86_64 > In that case, the link likely is "addrgenmode eui64". Note that this will > only restore IPv6 link local addresses. All other IPv6 addresses are gone -- > arguably, kernel (depending on sysctl configuration) might do autoconf again > and receive the same prefixes again. But manually added addresses are lost > either way. Exactly same as you said. > I would argue this works as expected. Only thing that surprises me is the > claim that Fedora 27 and Ubuntu behave differently. That would need some > investigation/clarification. I agree that it works as expected after tried. As ubuntu setting is "addrgenmode eui64", I will not compare with it now. I double checked it on Fedora27 and RHEL8. As RHEL8 nightly build also behave differently. Could you help confirm whether I need to file a bug to RHEL8 project? Thanks Below are my steps: [root@dhcp-0-105 ~]# ip -d link show dev eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000 link/ether 00:16:3e:fb:78:35 brd ff:ff:ff:ff:ff:ff promiscuity 0 addrgenmode none numtxqueues 2 numrxqueues 2 gso_max_size 65536 gso_max_segs 65535 [root@dhcp-0-105 ~]# ip a show dev eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:16:3e:fb:78:35 brd ff:ff:ff:ff:ff:ff inet 10.66.0.105/22 brd 10.66.3.255 scope global noprefixroute dynamic eth0 valid_lft 86332sec preferred_lft 86332sec inet6 fe80::b564:8ca0:69c1:3479/64 scope link noprefixroute valid_lft forever preferred_lft forever [root@dhcp-0-105 ~]# ip link set eth0 down [root@dhcp-0-105 ~]# ip a show dev eth0 2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether 00:16:3e:fb:78:35 brd ff:ff:ff:ff:ff:ff inet 10.66.0.105/22 brd 10.66.3.255 scope global noprefixroute dynamic eth0 valid_lft 86325sec preferred_lft 86325sec [root@dhcp-0-105 ~]# ip link set eth0 up [root@dhcp-0-105 ~]# ip a show dev eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:16:3e:fb:78:35 brd ff:ff:ff:ff:ff:ff inet 10.66.0.105/22 brd 10.66.3.255 scope global noprefixroute dynamic eth0 valid_lft 86316sec preferred_lft 86316sec inet6 fe80::b564:8ca0:69c1:3479/64 scope link noprefixroute valid_lft forever preferred_lft forever > > Can you explain why you are setting the interface down in the first place? > What are you trying to achive? Thanks I set the link down during the iperf3 load running. To see whether there are any exceptions happen. IPv4 connection restored after link down and up, but IPv6 did not. And compared with Fedora27, the behavior is different, so I filed this bug. Thomas, Is anything expected here w.r.t RHEL 8? Wasn't able to tell based on Xiao's question in c#3. Thanks! Sushil as discussed, I think we see here expected behaviour, and I would close this as NOTABUG. Objections anyone? As here is expected behavior in RHEL7, I will file another bug against RHEL8 to track different behavior for tracking. Please feel free to close it. Thanks Filed a new bug 1601669 to track it on RHEL8. *** Bug 1601669 has been marked as a duplicate of this bug. *** (In reply to Xiao, Liang from comment #6) > As here is expected behavior in RHEL7, I will file another bug against RHEL8 > to track different behavior for tracking. Please feel free to close it. > Thanks Issues for NM are always fixed upstream first. For upstream, we are generally cautious to not break behaviour in a bad way. When we change behaviour, it is usually also done in a way that is suitable for RHEL minor releases too -- there are exceptions to that, where we carry small downstream patches. Anyway, I am of the opinion, that this is not a bug, but desired behaviour (which is up for debate, of course). However: it is not declared desired behaviour, based on the assumption that it wouldn't be suitable for rhel-7. Hence, cloning for rhel-8 is not necessary to solve this issue, so I closed bug 1601669 as duplicate. Still need to figure out, what the pros + cons here are, and the way forward. I still lean to NOTABUG. Thanks. TODO. I think NM could be improved to better handle this situation. On master, when the interface goes up again we call ip_merge_and_apply(commit=1) to reapply the IP configuration, but we fail to add IPv6 addresses because they were removed from the AC IP config in update_ext_ip_config(). The ndisc instance keeps running and re-adds addresses only when the next (unsolicited) RA is received. Ideally NM should be able to detect that addresses were removed not because the user did so but because the interface went down, and should re-add them immediately when the interface goes up. related: https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=f069c98cc95494891215ddf261661afb742744ba This solves a similar problem for IP routes... Fixed upstream. If the interface only has ipv6 local link address, it cannot be recovered after port down and up. But it works in RHEL8.0. Could you help check the result?
[root@dhcp-14-171 ~]# ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:16:3e:fb:78:35 brd ff:ff:ff:ff:ff:ff
inet 10.66.14.171/23 brd 10.66.15.255 scope global dynamic noprefixroute eth0
valid_lft 85994sec preferred_lft 85994sec
inet6 fe80::216:3eff:fefb:7835/64 scope link noprefixroute
valid_lft forever preferred_lft forever
[root@dhcp-14-171 ~]# ip link set dev eth0 down
[root@dhcp-14-171 ~]# ip link set dev eth0 up
[root@dhcp-14-171 ~]# ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:16:3e:fb:78:35 brd ff:ff:ff:ff:ff:ff
inet 10.66.14.171/23 brd 10.66.15.255 scope global dynamic noprefixroute eth0
valid_lft 86399sec preferred_lft 86399sec
If the port has 2 ipv6 addresses, both can be recovered now.
[root@vm-198-147 ~]# ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 10:13:24:35:45:2e brd ff:ff:ff:ff:ff:ff
inet 10.73.198.147/22 brd 10.73.199.255 scope global noprefixroute dynamic eth0
valid_lft 42618sec preferred_lft 42618sec
inet6 2620:52:0:49c4:1213:24ff:fe35:452e/64 scope global noprefixroute dynamic
valid_lft 2591948sec preferred_lft 604748sec
inet6 fe80::1213:24ff:fe35:452e/64 scope link noprefixroute
valid_lft forever preferred_lft forever
[root@vm-198-147 ~]# ip link set dev eth0 down
[root@vm-198-147 ~]# ip link set dev eth0 up
[root@vm-198-147 ~]# ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 10:13:24:35:45:2e brd ff:ff:ff:ff:ff:ff
inet 10.73.198.147/22 brd 10.73.199.255 scope global noprefixroute dynamic eth0
valid_lft 43199sec preferred_lft 43199sec
inet6 2620:52:0:49c4:1213:24ff:fe35:452e/64 scope global noprefixroute dynamic
valid_lft 2592000sec preferred_lft 604800sec
inet6 fe80::1213:24ff:fe35:452e/64 scope link noprefixroute
valid_lft forever preferred_lft forever
(In reply to Xiao, Liang from comment #15) > If the interface only has ipv6 local link address, it cannot be recovered > after port down and up. But it works in RHEL8.0. Could you help check the > result? Which NM version are you using? (In reply to Beniamino Galvani from comment #16) > (In reply to Xiao, Liang from comment #15) > > If the interface only has ipv6 local link address, it cannot be recovered > > after port down and up. But it works in RHEL8.0. Could you help check the > > result? > > Which NM version are you using? I am using the default NM in RHEL-7.7-20190502.1 build. [root@dhcp-14-158 ~]# rpm -qa|grep -i networkm NetworkManager-libnm-1.18.0-1.el7.x86_64 NetworkManager-team-1.18.0-1.el7.x86_64 NetworkManager-1.18.0-1.el7.x86_64 NetworkManager-config-server-1.18.0-1.el7.noarch NetworkManager-glib-1.18.0-1.el7.x86_64 NetworkManager-tui-1.18.0-1.el7.x86_64 This seems to work well:
12: eth10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 52:54:00:ed:7d:1c brd ff:ff:ff:ff:ff:ff
inet 10.19.189.159/23 brd 10.19.189.255 scope global noprefixroute dynamic eth10
valid_lft 86221sec preferred_lft 86221sec
inet6 2620:52:0:13bc:a800:7f95:46de:3901/64 scope global noprefixroute dynamic
valid_lft 2591968sec preferred_lft 604768sec
inet6 fe80::263:4c54:f831:848c/64 scope link noprefixroute
valid_lft forever preferred_lft forever
[root@wsfd-netdev34-vm-8 NetworkManager-ci]# ip link set eth10 down
[root@wsfd-netdev34-vm-8 NetworkManager-ci]# ip link set eth10 up
[root@wsfd-netdev34-vm-8 NetworkManager-ci]# ip a s eth10
12: eth10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 52:54:00:ed:7d:1c brd ff:ff:ff:ff:ff:ff
inet 10.19.189.159/23 brd 10.19.189.255 scope global noprefixroute dynamic eth10
valid_lft 86398sec preferred_lft 86398sec
inet6 2620:52:0:13bc:a800:7f95:46de:3901/64 scope global noprefixroute dynamic
valid_lft 2591999sec preferred_lft 604799sec
inet6 fe80::263:4c54:f831:848c/64 scope link noprefixroute
valid_lft forever preferred_lft forever
but this does not (as above comments said)
[root@wsfd-netdev34-vm-8 NetworkManager-ci]# ip a s eth9
11: eth9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 52:54:00:a9:a7:7c brd ff:ff:ff:ff:ff:ff
inet 192.168.100.216/24 brd 192.168.100.255 scope global noprefixroute dynamic eth9
valid_lft 3599sec preferred_lft 3599sec
inet6 fe80::470b:30db:ab2c:e918/64 scope link tentative noprefixroute
valid_lft forever preferred_lft forever
[root@wsfd-netdev34-vm-8 NetworkManager-ci]# ip link set dev eth9 down
[root@wsfd-netdev34-vm-8 NetworkManager-ci]# ip link set dev eth9 up
[root@wsfd-netdev34-vm-8 NetworkManager-ci]# ip a s eth9
11: eth9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 52:54:00:a9:a7:7c brd ff:ff:ff:ff:ff:ff
inet 192.168.100.216/24 brd 192.168.100.255 scope global noprefixroute dynamic eth9
valid_lft 3599sec preferred_lft 3599sec
Should be fixed by: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/commit/db46d9b82386fab3c139cde30698ca5ca827b948 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2302 |
Created a RHEL7.5 hvm guest residing Fedora27 xen4 host. NetworkManager service was started by default. network interface's ipv6 address missed after down and up it. Here is log; # ip address show dev eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 10:13:24:36:45:57 brd ff:ff:ff:ff:ff:ff inet 10.66.0.124/22 brd 10.66.3.255 scope global noprefixroute dynamic eth0 valid_lft 86263sec preferred_lft 86263sec inet6 fe80::1213:24ff:fe36:4557/64 scope link noprefixroute valid_lft forever preferred_lft forever # ip link set dev eth0 down # ip address show dev eth0 2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN group default qlen 1000 link/ether 10:13:24:36:45:57 brd ff:ff:ff:ff:ff:ff inet 10.66.0.124/22 brd 10.66.3.255 scope global noprefixroute dynamic eth0 valid_lft 86228sec preferred_lft 86228sec # ip link set dev eth0 up - No ipv6 address after interface up again. # ip address show dev eth0 2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 10:13:24:36:45:57 brd ff:ff:ff:ff:ff:ff inet 10.66.0.124/22 brd 10.66.3.255 scope global noprefixroute dynamic eth0 valid_lft 86398sec preferred_lft 86398sec Version-Release number of selected component (if applicable): Host: Fedora27 xen-4.9.1 RHEL Version: RHEL7.5(3.10.0-855.el7.x86_64) How reproducible: 100% Steps to Reproduce: 1. Create a hvm guest residing on Fedora27 xen4 host. 2. down and up eth0 in guest. #ip link set dev eth0 down #ip link set dev eth0 up 3. Check ipv6 address of eth0 # ip address show dev eth0 Actual results: No ipv6 address found Expected results: Interface up with ipv6 assigned. Additional info: - Cannot reproduce it on Fedora27 and Ubuntu. - Cannot reproduce it if only enable network.service - Can reproduce this bug with RHEL7.4 GA release installed, so it is not a regression. - Only can reproduce it if enable NetworkManager service - Can reproduce it in bare metal system(tg3 nic driver), so it is not a xen related bug - Can reproduce it both using "ip" or "ifconfig" to down and up network interface.