Bug 1696881
| Summary: | systemctl restart NetworkManager causes duplicate rules loading | ||
|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Curtis Taylor <cutaylor> |
| Component: | NetworkManager | Assignee: | sushil kulkarni <sukulkar> |
| Status: | CLOSED ERRATA | QA Contact: | Desktop QE <desktop-qa-list> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 7.6 | CC: | atragler, bgalvani, bpawade, fgiudici, lrintel, msugaya, pasik, rkhan, sukulkar, surkumar, thaller, vbenes |
| Target Milestone: | rc | ||
| Target Release: | --- | ||
| Hardware: | All | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Fixed In Version: | NetworkManager-1.18.4-1.el7 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-03-31 20:07:59 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Curtis Taylor
2019-04-05 20:58:23 UTC
When NetworkManger is restarted, /etc/NetworkManager/dispatcher.d/10-ifcfg-rh-routes.sh pre-up reports "RTNETLINK answers: File exists", presumably when it attempted to add a duplicate route. Apr 05 17:02:18 rhel7u5-1 nm-dispatcher[18901]: req:10 'pre-up' [eth1]: new request (1 scripts) Apr 05 17:02:18 rhel7u5-1 nm-dispatcher[18901]: req:7 'connectivity-change': start running ordered scripts... Apr 05 17:02:18 rhel7u5-1 dhclient[18929]: bound to 192.168.100.210 -- renewal in 1787 seconds. Apr 05 17:02:18 rhel7u5-1 nm-dispatcher[18901]: RTNETLINK answers: File exists Apr 05 17:02:18 rhel7u5-1 NetworkManager[18892]: <info> [1554498138.1561] device (eth1): state change: ip-check -> secondaries (reason 'none', sys-iface-state: 'assume') Apr 05 17:02:18 rhel7u5-1 NetworkManager[18892]: <info> [1554498138.1563] device (eth1): state change: secondaries -> activated (reason 'none', sys-iface-state: 'assume') Apr 05 17:02:18 rhel7u5-1 nm-dispatcher[18901]: req:8 'up' [ens12]: start running ordered scripts... Apr 05 17:02:18 rhel7u5-1 NetworkManager[18892]: <info> [1554498138.1673] device (eth1): Activation: successful, device activated. Apr 05 17:02:18 rhel7u5-1 NetworkManager[18892]: <info> [1554498138.1676] manager: startup complete Apr 05 17:02:18 rhel7u5-1 nm-dispatcher[18901]: req:11 'up' [eth1]: new request (3 scripts) The duplicate (or triplicate because a rule is added everytime NetworkManager is restarted) rule that is added causes the older rule(s) not to go away when the network is stopped: # ip rule show all | grep " 5" 32760: from 192.168.100.121 lookup 5 32761: from all iif eth1 lookup 5 32762: from 192.168.100.121 lookup 5 32763: from all iif eth1 lookup 5 32764: from 192.168.100.121 lookup 5 32765: from all iif eth1 lookup 5 # systemctl stop network >>>> Two rules are left # ip rule show all | grep " 5" 32762: from 192.168.100.121 lookup 5 32763: from all iif eth1 lookup 5 32764: from 192.168.100.121 lookup 5 32765: from all iif eth1 lookup 5 # ip add show eth1 4: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000 link/ether 52:54:00:27:c4:a5 brd ff:ff:ff:ff:ff:ff # systemctl start network >>>>> A third rule comes back # ip rule show all | grep " 5" 32760: from 192.168.100.121 lookup 5 32761: from all iif eth1 lookup 5 32762: from 192.168.100.121 lookup 5 32763: from all iif eth1 lookup 5 32764: from 192.168.100.121 lookup 5 32765: from all iif eth1 lookup 5 The first thing to note is that policy rules are global and independent of a device. While they look similar to routes, routes are always tied to an interface. So, when NetworkManager configures an interface, it can just reset all routes on that interface. For rules that is more complicated, because a rule could be there for several reasons. For example, multiple connection profiles may configure the same rule. Assume all these profiles are activated at the same time. When you start deactivating these profiles one by one, then the rule must be removed only when the last of these profiles gets deactivated (NetworkManager-dispatcher-routing-rules would get this wrong). Another example, after a restart or when the user issues `ip rule add` manually, NetworkManager has a problem to understand *why* the rule is there. Until now, NetworkManager completely ignores rules (aside NetworkManager-dispatcher-routing-rules). Note that in rhel-7.7, bug 1652653 will be fixed and NetworkManager natively supporting routing rules (not via the dispatcher scripts). Then this becomes a problem for NetworkManager (e.g. https://cgit.freedesktop.org/NetworkManager/NetworkManager/tree/src/devices/nm-device.c?id=a77c5d18c4e9838c5eb58b9e759da95481a8e298#n6481). --- When you stop NetworkManager ("restart" is the same as stop+start), then NetworkManager leaves the interface up (intentionally). In particular it does not call the dispatcher script to remove the rules/routes. I agree, the "RTNETLINK answers: File exists" message is for the routes. `ip route add` which fails if the route already exists. Note that the rules in rule-eth1 don't specify a "priority", that means, kernel automatically selects one, so you get: > 32764: from 192.168.100.121 lookup 5 > 32765: from all iif eth1 lookup 5 and if you call the same `ip rule add` command again, you'll get additional, different rules: > 32762: from 192.168.100.121 lookup 5 > 32763: from all iif eth1 lookup 5 > 32764: from 192.168.100.121 lookup 5 > 32765: from all iif eth1 lookup 5 Note that when you issue `ip rule remove iif eth1 table 5`, then kernel will treat the omitted priority as a wildcard and just kill one of the rules. In other words: the rule's priority is part of the identity (primary key) of the rule. If you omit the priority during `ip rule add`, then this part of the identity is determined by the kernel. You say "Rules do not load if existing", but the file does not specify an exact rule, so it's not clear whether it exists. It would be very complex trying to let /etc/NetworkManager/dispatcher.d/10-ifcfg-rh-routes.sh parse the command line to determine whether this will add a duplicate or not. --- There is no easy solution to this, except why are you even doing this? There are few reasons why NetworkManager.service should ever be restarted. Possible reasons are: - if there is a serious bug in NetworkManager. But in that case, a restart is only a workaround and the issue/bug must be resolved instead. - on package upgrade. But whenever you upgrade the package (`yum upgrade`), the safe way is to do a full reboot. - modify certain settings in NetworkManager.conf or installing plugins that don't support being reloaded without restarting (NetworkManager doesn't support SIGHUP for them). In particular, you should not restart NetworkManager to have changes to the network take effect. Restarting NetworkManager *should not* do any changes to the networking configuration. Of course, that is not entirely true, for example DHCP needs to be restarted which may have visible changes. But overall there should be as little changes as possible. So when you restarting NetworkManager to change the network configuration, NeworkManager tries to hard not to do any changes. Use `nmcli connection up $PROFILE` instead. Of course you may restart NetworkManager if you know what you are doing and are prepared to manually fix issues like this one. Note that with NetworkManager it is generally not best to call `systemctl restart network.service` either. This iterates over all ifcfg files, and calls `nmcli connection load && nmcli connection down && nmcli connection up` for all ifcfg files. In that sense, it's better than `systemctl restart NetworkManager.service. But it's a crude way of reconfiguring the network. Instead, use `nmcli connection up` (possibly `nmcli connection load` after editing files on disk). NetworkManager-dispatcher-routing-rules provides a dispatcher script /etc/NetworkManager/dispatcher.d/10-ifcfg-rh-routes.sh. It's a simple script, similar to calling /etc/sysconfig/network-scripts/ifup-routes and /etc/sysconfig/network-scripts/ifdown-routes. You can always find ways how to confuse the script. NetworkManager itself is not aware of what this dispatcher script does, nor it is aware of rules (not until rhel-7.7, bug 1652653). --- With initscripts scripts alone (no NetworkManager involved), if you call `ifup eth1` twice (omitting `ifdown` in between), you get the same problem. I guess that's why calling `ifup` / `ifdown` does not work well (with initscripts) and the suggested solution is to issue a full `systemctl restart network.service` or even better: reboot. Note that if you are calling `systemctl restart network.service` because you edited the rule-eth1 file, and you'd like to have the changes take effect, then this also won't work: # edit rule-eth1 to remove a rule sed -i 's/iif eth1 table 5/iif eth1 table 6/' /etc/sysconfig/network-scripts/rule-eth1 systemctl restart network.service # now old rule is still there, because ifdown-routes wasn't aware it. So, in that case, you'd also need to take care: /etc/sysconfig/network-scripts/ifdown-routes eth1 sed -i 's/iif eth1 table 5/iif eth1 table 6/' /etc/sysconfig/network-scripts/rule-eth1 systemctl restart network.service --- Workarounds: - don't restart NetworkManager in the first place. - specify a priority for the rules. Then multiple `ip rule add` commands will fail with "RTNETLINK answers: File exists" - don't use /etc/NetworkManager/dispatcher.d/10-ifcfg-rh-routes.sh but use your own script to configure rules that works better. The dispatcher script is there in the hope to work reasonably well for you. If you know better how to handle this, be welcome to use your improved script. - if you really want to use /etc/NetworkManager/dispatcher.d/10-ifcfg-rh-routes.sh *and* restart NetworkManager then be prepared to perform certain steps to workaround the issue: For example nmcli networking off systemctl restart NetworkManager nmcli networking on or nmcli device disconnect eth1 systemctl restart NetworkManager nmcli device connect eth1 or systemctl stop NetworkManager /etc/sysconfig/network-scripts/ifdown-routes eth1 systemctl start NetworkManager I would close this issue as "CANTFIX". TL;DR: always specify a unique priority for all your rules. Thomas, Thank you! I agree with almost everything in your update, especially the caveat about why even restart NetworkManager in the first place. However, even when specifying the priority in the ifcfg-eth1 file we still get duplicates. # uname -a Linux rhel7u5-1 3.10.0-957.10.1.el7.x86_64 #1 SMP Thu Feb 7 07:12:53 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux # ip rule show 0: from all lookup local 200: from all iif eth1 lookup 5 201: from 192.168.100.121 lookup 5 32766: from all lookup main 32767: from all lookup default # systemctl restart NetworkManager # ip rule show 0: from all lookup local 200: from all iif eth1 lookup 5 200: from all iif eth1 lookup 5 <--- duplicate shows up, not file exists error 201: from 192.168.100.121 lookup 5 201: from 192.168.100.121 lookup 5 <--- duplicate shows up, not file exists error 32766: from all lookup main 32767: from all lookup default # cat rule-eth1 priority 200 iif eth1 table 5 priority 201 from 192.168.100.121 table 5 # ifdown eth1 Device 'eth1' successfully disconnected. # ip rule show 0: from all lookup local 200: from all iif eth1 lookup 5 <--- only one priority 200 rule was deleted 201: from 192.168.100.121 lookup 5 <--- only one priority 201 rule was deleted 32766: from all lookup main 32767: from all lookup default The kernel didn't throw a "file exists" error. Do you think this is a separate Bug? > However, even when specifying the priority in the ifcfg-eth1 file we still get duplicates.
oh, that's very odd.
Fedora 29:
# uname -a
Linux rh1 5.0.5-200.fc29.x86_64 #1 SMP Wed Mar 27 20:58:04 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
# ip netns del t &>/dev/null; ip netns add t
# ip --netns t -d rule add priority 100 from all iif eth1 lookup 5
# ip --netns t -d rule add priority 100 from all iif eth1 lookup 5
RTNETLINK answers: File exists
rhel-7.6:
# uname -a
Linux ip-172-31-30-201.us-east-2.compute.internal 3.10.0-957.el7.x86_64 #1 SMP Thu Oct 4 20:48:51 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
# ip --netns t -d rule add priority 100 from all iif eth1 lookup 5
# ip --netns t -d rule add priority 100 from all iif eth1 lookup 5
# ip --netns t -d rule
0: from all lookup local
100: from all iif eth1 [detached] lookup 5
100: from all iif eth1 [detached] lookup 5
32766: from all lookup main
32767: from all lookup default
This seems like a different issue to me (kernel related).
> for the same customer, I suggested below commands to restart NetworkManager: (as per #3)
This does not seem related to this issue. Could you instead open a new bugzilla?
Also, as `nmcli co up eth0.10` fails, can you attach the relevant portion of the journal/syslog? Preferably with level=TRACE log level.
Thank you.
> > However, even when specifying the priority in the ifcfg-eth1 file we still get duplicates. > > oh, that's very odd. Opened kernel bug 1700691 for this. I think there is nothing left to do here. If the rules file does not specify a rule priority, the upon each call of ifup (which issues `ip rule add`), a rule with a new, unique priority will be added. As NetworkManager leaves the interface up when exiting (and not calling ifdown to delete it), upon restart a duplicate rule will be added. The "solution" is first of all not to leave the priority unspecified and explicitly specify the rule. It anyway seems error prone, because if you call `ifup` multiple times, you keep adding the same rule (with different automatic priorities). Also, restarting NetworkManager can be problematic and should seldom be necessary. If you do so, then be prepared to handle this yourself (e.g. by flushing the rules that you want). --- On recent versions of NetworkManager, you can also configure rules in NetworkManager directly, contrary to ifcfg's "rule" files which are handled by a dispatcher scripts. So, you might consider to use that instead: e.g. nmcli connection modify "$PROFILE" +ipv4.routing-rules "priority 200 iif eth1 table 5" That should work well (if you have issues with that, please open a separate bug however to keep the issue focused). Note that rhel-7.8 will slightly improve NetworkManager's native handling of routing rules, but it should work on rhel-7.7 fine too. --- Kernel had bug 1700691, that it would allow adding the exact same rule multiple times. That results in duplicate rules if you call ifup multiple times -- even if the priority was explicitly specified. Kernel would do so even if user-space specified NLM_F_EXCL flag. That was fixed in kernel in the meantime, and NetworkManager for a short time used not to set this flag either (that is fixed now as well). Note that kernel still adds duplicate rules if user-space doesn't set NLM_F_EXCL flag. I consider that a serious issue, but upstream rejected to fix that because it would break user space applications. I don't think there is anything left to do here, and I don't think there will be any further fixes necessary than what is already in rhel-7.7. Moving to MODIFIED, it will be later added to rhel-7.8 errata for QA. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:1162 > In one of the service requests, we see the same issue(systemctl restart NetworkManager causes duplicate rules loading) with the latest NetworkManager version. Sharing the reproduction steps, kindly check and share your observation. You are commenting on a bug that is marked as fixed for two years. I also tried to explain that this is the expected outcome (and why). > # cat rule-ens3 > from 192.168.122.0/24 table 10 > to 10.181.x.xxx/27 table 10 If you are using a rule file, then it's not NetworkManager who configures the routing rules. Instead, it's the dispatcher script /usr/lib/NetworkManager/dispatcher.d/10-ifcfg-rh-routes.sh from NetworkManager-dispatcher-routing-rules package. Basically, what I already said applies. I would suggest to configure the rules in NetworkManager, and not in a rule file (see comment 11). Or alternatively, don't restart NetworkManager. Or alternatively, understand what /usr/lib/NetworkManager/dispatcher.d/10-ifcfg-rh-routes.sh does and configure the rules accordingly (to specify at least a priority). Or alternatively, fix the rules yourself (with a shell script) when calling `systemctl restart NetworkManager`. |