(In reply to Thomas Haller from comment #0)
> _ip route append 7.7.7.0/24 dev net1
> _ip -4 route append  local default dev net1 table 10223
> _ip -4 route append  192.168.4.0/24 table 10223 nexthop via 7.7.7.1 dev net1

This one added a route with struct fib_nh fi->fib_nh_scope 254 (RT_SCOPE_HOST/local address)

> _ip -4 route prepend default dev net1 proto kernel table 10223
> 
> _ip -d -4 route show table all
> 
> _ip -4 route append  192.168.4.0/24 table 10223 nexthop via 7.7.7.1 dev net1

This one added a route with struct fib_nh fi->fib_nh_scope 253 (RT_SCOPE_LINK/direct route)

The fib_nh_scope was updated in

- fib_create_info
  - fib_check_nh -> the cfg->fc_scope is 0, which is RT_SCOPE_UNIVERSE/global
    - fib_check_nh_v4_gw
      - fib_table_lookup

As there is a local default route, we got scope 254 for the first nexthop route adding.
And then we prepend a direct default route, we got scope 253 the second time.
The scope info is stored in struct fib_nh nh->fib_nh_scope

But when dump the route in fib_dump_info(). The rtm->rtm_scope info is got from struct fib_info fi->fib_scope, which is 0 (RT_SCOPE_UNIVERSE/global).
So these 2 routes look the same.

This issue is similar to bug 2162315. The nexthop's weight and scope are meaningless for a non-multipath route. So the kernel is not export this to userspace. Upstream seems no intend to fix/export them. They suggest using the new nexthop api or trying what libnl does to resolve the route cache issue.

For the new nexthop api, it uses unique nhid and does not has this scope issue as it only exists with legacy nexhop api:

# ip nexthop add id 1 via 172.16.104.100 dev dummy1
# ip route add local default dev dummy1 table 200
# ip route add 172.16.107.0/24 table 200 nhid 1
# ip route prepend default dev dummy1 table 200
# ip route append 172.16.107.0/24 table 200 nhid 1     <- this will fail as same route has exists
RTNETLINK answers: File exists

# ip route show table 200
default dev dummy1 scope link
local default dev dummy1 scope host
172.16.107.0/24 nhid 1 via 172.16.104.100 dev dummy1

# ip nexthop add id 2 via 172.16.104.100 dev dummy1     <- if you still want to add same nexthop, you need use diff nhid
# ip route append 172.16.107.0/24 table 200 nhid 2
# ip route show table 200
default dev dummy1 scope link
local default dev dummy1 scope host
172.16.107.0/24 nhid 1 via 172.16.104.100 dev dummy1
172.16.107.0/24 nhid 2 via 172.16.104.100 dev dummy1

(In reply to Hangbin Liu from comment #2)
> The nexthop's weight and scope are meaningless for a non-multipath route.

The reproducer from comment 0 uses no weight/scope. 

Two identical `ip route append` commands result in two routes, that look identical to user space. That kernel internally gets confused about the scope, that's the bug.

> So the kernel is not export this to userspace.

Even if kernel would export the difference to user space, it seems the effect is already wrong (making the same `ip route append` command resulting in different routes, depending on the existence of another route).


> They suggest using
> the new nexthop api or trying what libnl does to resolve the route cache
> issue.

What exactly does libnl do there?

> For the new nexthop api

Seems unrelated. The problem is not that NetworkManager may do this. The problem is that any user can type up the commands from comment 0 in their terminal, and cause a wrong view of the world in NetworkManager. That's regardless, whether NetworkManager would use nexthop objects or just ever avoiding to get into such a situation on its own.

Issue migration from Bugzilla to Jira is in process at this time. This will be the last message in Jira copied from the Bugzilla bug.

This BZ has been automatically migrated to the issues.redhat.com Red Hat Issue Tracker. All future work related to this report will be managed there.

Due to differences in account names between systems, some fields were not replicated.  Be sure to add yourself to Jira issue's "Watchers" field to continue receiving updates and add others to the "Need Info From" field to continue requesting information.

To find the migrated issue, look in the "Links" section for a direct link to the new issue location. The issue key will have an icon of 2 footprints next to it, and begin with "RHEL-" followed by an integer.  You can also find this issue by visiting https://issues.redhat.com/issues/?jql= and searching the "Bugzilla Bug" field for this BZ's number, e.g. a search like:

"Bugzilla Bug" = 1234567

In the event you have trouble locating or viewing this issue, you can file an issue by sending mail to rh-issues. You can also visit https://access.redhat.com/articles/7032570 for general account information.