|Summary:||"ipv4.route-table" is reset to value 0 after reboot, if rule- file is present|
|Product:||Red Hat Enterprise Linux 7||Reporter:||Mai Ling <mailinglists35>|
|Component:||NetworkManager||Assignee:||sushil kulkarni <sukulkar>|
|Status:||CLOSED DUPLICATE||QA Contact:||Desktop QE <desktop-qa-list>|
|Version:||7.5||CC:||atragler, bgalvani, fgiudici, lrintel, pasik, ptalbert, rkhan, sukulkar, thaller|
|Fixed In Version:||Doc Type:||If docs needed, set a value|
|Doc Text:||Story Points:||---|
|Last Closed:||2020-01-07 15:05:59 UTC||Type:||Bug|
|oVirt Team:||---||RHEL 7.3 requirements from Atomic Host:|
|Cloudforms Team:||---||Target Upstream Version:|
|Bug Depends On:||1652653|
Description Mai Ling 2018-06-09 04:31:19 UTC
Description of problem: If a connection has a custom ipv4.route-table value other than zero, after reboot is reset to zero and routes are created in table main instead of table number. Version-Release number of selected component (if applicable): NetworkManager-1.10.2-14.el7_5.x86_64 How reproducible: Always Steps to Reproduce: nmcli con delete myconn nmcli con delete bridge-slave-p4p1-myconn rm -f /etc/sysconfig/network-scripts/rule-myconn nmcli connection add type bridge connection.id myconn connection.interface-name myconn nmcli connection add type ethernet slave-type bridge master myconn ifname p4p1 connection.id bridge-slave-p4p1-myconn nmcli connection modify myconn ipv6.method ignore nmcli connection modify myconn ipv4.addresses 10.1.40.250/24 nmcli connection modify myconn ipv4.method manual nmcli connection modify myconn ipv4.route-table 100 nmcli connection modify myconn ipv4.gateway 10.1.40.254 nmcli connection modify myconn ipv4.dns-search myconn nmcli connection modify myconn ipv4.dns-options edns0 nmcli conn down myconn cat /etc/sysconfig/network-scripts/rule-myconn1 > /etc/sysconfig/network-scripts/rule-myconn nmcli con up myconn [root@gw2 ~]# cat /etc/sysconfig/network-scripts/rule-myconn1 iif p4p1 table 100 from 10.1.40.250 table 100 iif lo table 100 [root@gw2 ~]# ip ru 0: from all lookup local 32763: from all iif lo lookup 100 32764: from 10.1.40.250 lookup 100 32765: from all iif p4p1 lookup 100 32766: from all lookup main 32767: from all lookup default [root@gw2 ~]# ip ro 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 [root@gw2 ~]# ip ro s t 100 default via 10.1.40.254 dev myconn proto static metric 427 10.1.40.0/24 dev myconn proto kernel scope link src 10.1.40.250 metric 427 REBOOT and re-run 'ip ru' and 'ip ro' Actual results: rules are ok but routes are not put in their table but instead into main; additionally I see a strange metric value. 'nmcli con show' shows value 0 instead of 100 for ipv4.route-table even if ifcfg-myconn has "IPV4_ROUTE_TABLE=100" ([root@gw2 ~]# nmcli -g ipv4.route-table con show myconn 0 [root@gw2 ~]#) [root@gw2 ~]# ip ru 0: from all lookup local 32763: from all iif lo lookup 100 32764: from 10.1.40.250 lookup 100 32765: from all iif p4p1 lookup 100 32766: from all lookup main 32767: from all lookup default [root@gw2 ~]# ip ro default via 10.1.40.254 dev myconn proto static metric 426 10.1.40.0/24 dev myconn proto kernel scope link src 10.1.40.250 metric 426 192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 [root@gw2 ~]# ip ro s t 100 [root@gw2 ~]# Expected results: see the same "ip ru" and "ip ro" output after reboot Additional info: KB 288823 updated 22 may says I should do this using exclusively rule- and route- files. But then why do I have ipv4.route-table option and what's the use if I can't use it with ifcfg rules or natively? I'd like to avoid editing files and only use nmcli for everything! Or at least make it work partially by not resetting the ipv4.route-table at boot and live in peace with rule- files!
Comment 2 Mai Ling 2018-06-09 04:40:54 UTC
also issue #1384799 does not seem to have effect, as it still gives me the message "Cannot modify a connection that has an associated 'rule-' or 'rule6-' file" when I attempt to fix ipv4.route-table with nmcli after boot.
Comment 3 Thomas Haller 2018-06-11 08:34:11 UTC
It's a missing feature. Traditionally, NetworkManager did not support specifying a routing-table at all. For that, there is a workaround in form of a dispatcher script  . The dispatcher script becomes active, when a rule- / rule6- file exists, and it calls ifup-routes command accordingly. That is, the dispatcher script calls ifup-routes, but that cannot only handle rules, it also handles route files. For that reason, when you configure a rule- file, it is assumed you are using initscripts to manage the rules and the routes. For that reason, the presence of a rule-file prevents NetworkManager from configuring any static-routes (ipv4.routes) or a routing table (ipv4.route-table). This limitation was present for a long time, it was slightly relaxed with bug 1384799, but it is still essentially there. Yes, this results in a catch22, because NetworkManager currently does not support configuring routing rules, so users are required to configure them themselvs. But users cannot use initscripts for that, because than NetworkManager does not handle static routes or ipv4.route-table. The long term solution is to natively support routing rules in NetworkManager. The restriction that we see exists, is so that later when we introduce support for this, we won't change existing behaviour. The short term workarounds are: - don't rely on NetworkManager's ipv4.route-table and do it all with initscripts (writing rule- and route- files). - create the rules any other way, for example with a dispatcher scripts. If you look at /etc/sysconfig/network-scripts/ifup-routes, it's no magic.  NetworkManager-dispatcher-routing-rules.rpm package  /etc/NetworkManager/dispatcher.d/10-ifcfg-rh-routes.sh
Comment 4 Thomas Haller 2018-06-13 11:47:17 UTC
> "ipv4.route-table" is reset to value 0 after reboot, if rule- file is present btw, what the subject describes, is correct but misleading. What happens is: 1) first, user creates profile with ipv4.route-table=42 2) then, user creates rule-file. Unless user issues `nmcli connection load $FILE` or `nmcli connection reload`, creating files has no immediate effects to NetworkManager. 3) after reboot, the profile is interpreted differently, with ipv4.route-table=0 (due to the now-present rule-file). I think, the way forward here is to add native support to rules to NetworkManager. Which is however a large effort. If we agree, I would re-purpose this bug as feature request for supporting routing rules.
Comment 5 Mai Ling 2018-06-27 15:32:23 UTC
(In reply to Thomas Haller from comment #4) > > "ipv4.route-table" is reset to value 0 after reboot, if rule- file is present > > btw, what the subject describes, is correct but misleading. > > What happens is: > > 1) first, user creates profile with ipv4.route-table=42 > 2) then, user creates rule-file. Unless user issues `nmcli connection load > $FILE` or `nmcli connection reload`, creating files has no immediate effects > to NetworkManager. agree but: in my described scenario, issuing `nmcli connection up` with the rule file present _does_ setup rules correctly (review the `ip ru` and `ip ro` output right after `nmcli c up`)! it breaks only after reboot - but running the commands on a live system it works immediately. > 3) after reboot, the profile is interpreted differently, with > ipv4.route-table=0 (due to the now-present rule-file). > > > > I think, the way forward here is to add native support to rules to > NetworkManager. Which is however a large effort. If we agree, I would > re-purpose this bug as feature request for supporting routing rules. *full* ip rule + rt_tables support for n-m would be awesome (so no more n-m dispatcher helper for rule- files and no more manual editing of rt_tables - if I understand correctly when we say "native" support). but until then, I'd be happy just to have the running config work after reboot.
Comment 6 Thomas Haller 2018-11-22 14:38:03 UTC
RFE for adding native support for routing rules to NetworkManager: bug 1652653
Comment 8 Thomas Haller 2020-01-07 15:05:59 UTC
> agree but: in my described scenario, issuing `nmcli connection up` with the rule file present _does_ setup rules correctly (review the `ip ru` and `ip ro` output right after `nmcli c up`)! it breaks only after reboot - but running the commands on a live system it works immediately. If I understand correctly, that is because as long as you don't reload the file, NetworkManager is not aware of the rule file. On the other hand, the dispatcher scripts is aware of the rule file. So, in that case NM configures routes while also the dispatcher script configures them. After reboot (or `nmcli con reload`) NetworkManager sees the rule file. This causes the profile in NetworkManager to have no routes/rules (because those are assume to be handled by the dispatcher scripts). Then it works as expected. The problem is that the file was modified on disk without telling NetworkManager. I don't see anything unexpected here -- aside that of course this legacy behavior with the dispatcher script is not stellar. But it's not clear how to fix it without moving away from the dispatcher script completely. And that is what happened... > I'd like to avoid editing files and only use nmcli for everything! Or at least make it work partially by not resetting the ipv4.route-table at boot and live in peace with rule- files! Yes, this is now possible as NetworkManager gained full support for routing rules. Note that it will not store rules in the "rule-*" file, because those are reserved for the legacy mode of operation. So, either use nmcli to edit rules () or use the new style format (if you wish to edit files).  e.g. nmcli connection modify "$PROFILE" +ipv4.routing-rules "priority 10 from 192.168.6.24/32 table 66" (the syntax for rules follows iproute2's syntax). I will close this bug as duplicate of bug 1652653. Usually, I would move the bug to ON-QA for verification, but I don't think there are any special parts that aren't yet covered by bug 1652653. Please open a new bug if something is not working as expected. *** This bug has been marked as a duplicate of bug 1652653 ***