It was pointed out by nmstate team that the 'next-hop-address': '::' is probably an issue and not the fact about having different metric. So this error message is misleading. 
Can you please share supervdsm.log part that shows which parameters were passed from engine?

This error seems to be fixed on nmstate-0.3 and nmstate-0.4. I am investigating why it is failing on nmstate-0.2. I've pointed out to Ales that the next-hop-address: '::' could be an issue and causing the gateway issue on nmstate-0.2). I will report more details soon. Thank you!

Yes, this nmstate-0.2.10-1.el8.noarch and NetworkManager-1.22.8-5.el8_2.x86_64

I actually did a better investigation than the above, but lost the text file of my notes :(
I'll try to recreate some of it here.

That NmstateNotImplementedError is raised on a function it checks if it has gateway configured
on any of the interfaces, and checks if any default route (0.0.0.0/0) on the routes.

def add_routes(setting_ip, routes):
    for route in routes:
        if route[Route.DESTINATION] in (
            IPV4_DEFAULT_GATEWAY_DESTINATION,
            IPV6_DEFAULT_GATEWAY_DESTINATION,
        ):
            if setting_ip.get_gateway():
                raise NmstateNotImplementedError(
                    "Only a single default gateway is supported due to a "
                    "limitation of NetworkManager: "
                    "https://bugzilla.redhat.com/1707396"
                )
            _add_route_gateway(setting_ip, route)
        else:
            _add_specfic_route(setting_ip, route)

The metric is definitely involved, the errors does not happen if the metric is -1.

From the code I thought NetworkManager would merge both routes if the metric is set to auto (-1).
But I didn't get as far as confirming it, this was the related code I reached.

nm_ip4_config_merge_setting (NMIP4Config *self,
                             NMSettingIPConfig *setting,
                             NMSettingConnectionMdns mdns,
                             NMSettingConnectionLlmnr llmnr,
                             guint32 route_table,
                             guint32 route_metric)
{
        /* Gateway */
        if (   !nm_setting_ip_config_get_never_default (setting)
            && (gateway_str = nm_setting_ip_config_get_gateway (setting))
            && inet_pton (AF_INET, gateway_str, &gateway_bin) == 1
            && gateway_bin) {
                const NMPlatformIP4Route r = {
                        .rt_source = NM_IP_CONFIG_SOURCE_USER,
                        .gateway = gateway_bin,
                        .table_coerced = nm_platform_route_table_coerce (route_table),
                        .metric = route_metric,
                };

                _add_route (self, NULL, &r, NULL);   <----------------
        }

...

        /* Routes */
        for (i = 0; i < nroutes; i++) {
                NMIPRoute *s_route = nm_setting_ip_config_get_route (setting, i);
                NMPlatformIP4Route route;

                if (nm_ip_route_get_family (s_route) != AF_INET) {
                        nm_assert_not_reached ();
                        continue;
                }

                memset (&route, 0, sizeof (route));
                nm_ip_route_get_dest_binary (s_route, &route.network);

                route.plen = nm_ip_route_get_prefix (s_route);
                nm_assert (route.plen <= 32);

                nm_ip_route_get_next_hop_binary (s_route, &route.gateway);
                if (nm_ip_route_get_metric (s_route) == -1)
                        route.metric = route_metric;
                else
                        route.metric = nm_ip_route_get_metric (s_route);
                route.rt_source = NM_IP_CONFIG_SOURCE_USER;

                route.network = nm_utils_ip4_address_clear_host_address (route.network, route.plen);

                _nm_ip_config_merge_route_attributes (AF_INET,
                                                      s_route,
                                                      NM_PLATFORM_IP_ROUTE_CAST (&route),
                                                      route_table);
                _add_route (self, NULL, &route, NULL);
        }

So my conclusion was that the routes would be merged if the metric of them is the same,
otherwise we get this fake duplicate route. But I could be wrong and well off here,
haven't debugged enough.

(In reply to Ales Musil from comment #3)
> It was pointed out by nmstate team that the 'next-hop-address': '::' is
> probably an issue and not the fact about having different metric. So this
> error message is misleading. 
> Can you please share supervdsm.log part that shows which parameters were
> passed from engine?

Will do, will also do a test with '::'...

Regarding  'next-hop-address': '::' 

I don't think this is it. First, from the traceback in comment #0, we see its in the ipv4 code path.

But I've checked it with IPv6 disabled, its reproducible, again the metric is the key aspect.

MainProcess|jsonrpc/6::INFO::2020-11-13 08:57:48,541::configurator::197::root::(_setup_nmstate) Desired state: {
'interfaces': [
  {'name': 'enp1s0', 'state': 'up', 'mtu': 1500, 'ipv4': {'enabled': False}, 'ipv6': {'enabled': False}}, 
  {'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500, 'bridge': {'port': [{'name': 'enp1s0'}], 'options': {'stp': {'enabled': False}}}, 
   'ipv4': {'enabled': True, 'address': [{'ip': '192.168.100.2', 'prefix-length': 24}], 'dhcp': False}, 
   'ipv6': {'enabled': False}}], 
'routes': 
  {'config': [{'next-hop-address': '192.168.100.254', 'next-hop-interface': 'ovirtmgmt', 'destination': '0.0.0.0/0', 'table-id': 0}]}, 'dns-resolver': {'config': {'server': ['192.168.100.254']}}}

MainProcess|jsonrpc/6::ERROR::2020-11-13 08:57:53,591::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper) Error in setupNetworks
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", line 95, in wrapper
    res = func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 241, in setupNetworks
    _setup_networks(networks, bondings, options, net_info)
  File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 266, in _setup_networks
    networks, bondings, options, net_info, in_rollback
  File "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", line 154, in setup
    _setup_nmstate(networks, bondings, options, in_rollback)
  File "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py", line 199, in _setup_nmstate
    nmstate.setup(desired_state, verify_change=not in_rollback)
  File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate/api.py", line 48, in setup
    state_apply(desired_state, verify_change=verify_change)
  File "/usr/lib/python3.6/site-packages/libnmstate/deprecation.py", line 40, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line 96, in wrapped
    ret = func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 73, in apply
    state.State(desired_state), verify_change, commit, rollback_timeout
  File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 159, in _apply_ifaces_state
    state2edit, original_desired_state
  File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 263, in _edit_interfaces
    iface2prepare + proxy_ifaces, original_desired_state
  File "/usr/lib/python3.6/site-packages/libnmstate/nm/applier.py", line 126, in prepare_edited_ifaces_configuration
    original_desired_iface_state=original_desired_iface_state,
  File "/usr/lib/python3.6/site-packages/libnmstate/nm/applier.py", line 422, in _build_connection_profile
    iface_desired_state.get(Interface.IPV4), base_profile
  File "/usr/lib/python3.6/site-packages/libnmstate/nm/ipv4.py", line 79, in create_setting
    setting_ipv4, config.get(nm_route.ROUTE_METADATA, [])
  File "/usr/lib/python3.6/site-packages/libnmstate/nm/route.py", line 166, in add_routes
    "Only a single default gateway is supported due to a "
libnmstate.error.NmstateNotImplementedError: Only a single default gateway is supported due to a limitation of NetworkManager: https://bugzilla.redhat.com/1707396

Ales, this was received from the engine:

2020-11-13 08:56:52,456+1000 INFO  (jsonrpc/6) [api.network] START setupNetworks(networks={'ovirtmgmt': {'netmask': '255.255.255.0', 'ipv6autoconf': False, 'nic': 'enp1s0', 'bridged': 'true', 'ipaddr': '192.168.100.2', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no', 'gateway': '192.168.100.254', 'mtu': 1500, 'switch': 'legacy'}}, bondings={}, options={'connectivityTimeout': 120, 'commitOnSuccess': True, 'connectivityCheck': 'true'}) from=::ffff:192.168.100.253,39084, flow_id=0e574aed-67dc-4b60-8574-9e881141f744 (api:48)

(In reply to Germano Veit Michel from comment #6)
> Regarding  'next-hop-address': '::' 
> 
> I don't think this is it. First, from the traceback in comment #0, we see
> its in the ipv4 code path.
> 
> But I've checked it with IPv6 disabled, its reproducible, again the metric
> is the key aspect.
> 
> MainProcess|jsonrpc/6::INFO::2020-11-13
> 08:57:48,541::configurator::197::root::(_setup_nmstate) Desired state: {
> 'interfaces': [
>   {'name': 'enp1s0', 'state': 'up', 'mtu': 1500, 'ipv4': {'enabled': False},
> 'ipv6': {'enabled': False}}, 
>   {'name': 'ovirtmgmt', 'type': 'linux-bridge', 'state': 'up', 'mtu': 1500,
> 'bridge': {'port': [{'name': 'enp1s0'}], 'options': {'stp': {'enabled':
> False}}}, 
>    'ipv4': {'enabled': True, 'address': [{'ip': '192.168.100.2',
> 'prefix-length': 24}], 'dhcp': False}, 
>    'ipv6': {'enabled': False}}], 
> 'routes': 
>   {'config': [{'next-hop-address': '192.168.100.254', 'next-hop-interface':
> 'ovirtmgmt', 'destination': '0.0.0.0/0', 'table-id': 0}]}, 'dns-resolver':
> {'config': {'server': ['192.168.100.254']}}}
> 
> MainProcess|jsonrpc/6::ERROR::2020-11-13
> 08:57:53,591::supervdsm_server::97::SuperVdsm.ServerCallback::(wrapper)
> Error in setupNetworks
> Traceback (most recent call last):
>   File "/usr/lib/python3.6/site-packages/vdsm/supervdsm_server.py", line 95,
> in wrapper
>     res = func(*args, **kwargs)
>   File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 241, in
> setupNetworks
>     _setup_networks(networks, bondings, options, net_info)
>   File "/usr/lib/python3.6/site-packages/vdsm/network/api.py", line 266, in
> _setup_networks
>     networks, bondings, options, net_info, in_rollback
>   File
> "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py",
> line 154, in setup
>     _setup_nmstate(networks, bondings, options, in_rollback)
>   File
> "/usr/lib/python3.6/site-packages/vdsm/network/netswitch/configurator.py",
> line 199, in _setup_nmstate
>     nmstate.setup(desired_state, verify_change=not in_rollback)
>   File "/usr/lib/python3.6/site-packages/vdsm/network/nmstate/api.py", line
> 48, in setup
>     state_apply(desired_state, verify_change=verify_change)
>   File "/usr/lib/python3.6/site-packages/libnmstate/deprecation.py", line
> 40, in wrapper
>     return func(*args, **kwargs)
>   File "/usr/lib/python3.6/site-packages/libnmstate/nm/nmclient.py", line
> 96, in wrapped
>     ret = func(*args, **kwargs)
>   File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line 73,
> in apply
>     state.State(desired_state), verify_change, commit, rollback_timeout
>   File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line
> 159, in _apply_ifaces_state
>     state2edit, original_desired_state
>   File "/usr/lib/python3.6/site-packages/libnmstate/netapplier.py", line
> 263, in _edit_interfaces
>     iface2prepare + proxy_ifaces, original_desired_state
>   File "/usr/lib/python3.6/site-packages/libnmstate/nm/applier.py", line
> 126, in prepare_edited_ifaces_configuration
>     original_desired_iface_state=original_desired_iface_state,
>   File "/usr/lib/python3.6/site-packages/libnmstate/nm/applier.py", line
> 422, in _build_connection_profile
>     iface_desired_state.get(Interface.IPV4), base_profile
>   File "/usr/lib/python3.6/site-packages/libnmstate/nm/ipv4.py", line 79, in
> create_setting
>     setting_ipv4, config.get(nm_route.ROUTE_METADATA, [])
>   File "/usr/lib/python3.6/site-packages/libnmstate/nm/route.py", line 166,
> in add_routes
>     "Only a single default gateway is supported due to a "
> libnmstate.error.NmstateNotImplementedError: Only a single default gateway
> is supported due to a limitation of NetworkManager:
> https://bugzilla.redhat.com/1707396
> 
> Ales, this was received from the engine:
> 
> 2020-11-13 08:56:52,456+1000 INFO  (jsonrpc/6) [api.network] START
> setupNetworks(networks={'ovirtmgmt': {'netmask': '255.255.255.0',
> 'ipv6autoconf': False, 'nic': 'enp1s0', 'bridged': 'true', 'ipaddr':
> '192.168.100.2', 'defaultRoute': True, 'dhcpv6': False, 'STP': 'no',
> 'gateway': '192.168.100.254', 'mtu': 1500, 'switch': 'legacy'}},
> bondings={}, options={'connectivityTimeout': 120, 'commitOnSuccess': True,
> 'connectivityCheck': 'true'}) from=::ffff:192.168.100.253,39084,
> flow_id=0e574aed-67dc-4b60-8574-9e881141f744 (api:48)

Thank you for checking that. I just wanted to make sure that we are clear in what exactly is causing this failure.

(In reply to Fernando F. Mancera from comment #4)
> This error seems to be fixed on nmstate-0.3 and nmstate-0.4. I am
> investigating why it is failing on nmstate-0.2. I've pointed out to Ales
> that the next-hop-address: '::' could be an issue and causing the gateway
> issue on nmstate-0.2). I will report more details soon. Thank you!

Any updates, or may I move the bug to nmstate?

Changing to nmstate bug.

With nmstate-0.3 and nmstate-0.2, Nmstate will raise exception regarding two gateways as nmstate think metric difference means different route.

With nmstate-0.4 and nmstate-1.0, Nmstata will not raise exception, but system will got two gateway each with their own metrics.

My intention on fix is: ignore metric difference.

Impact on this fix, user cannot create two identical route with only metric difference. It is allowed by kernel, but I don't see any real usage
on this.


Hi Dominik,

Do you need me backpoint fixes to RHEL 8.2 or/and 8.3? The workaround would be include a absent route for gateway in desire state also.

(In reply to Gris Ge from comment #10)
> Changing to nmstate bug.
> 
> With nmstate-0.3 and nmstate-0.2, Nmstate will raise exception regarding two
> gateways as nmstate think metric difference means different route.
> 
> With nmstate-0.4 and nmstate-1.0, Nmstata will not raise exception, but
> system will got two gateway each with their own metrics.
> 
> My intention on fix is: ignore metric difference.
> 
> Impact on this fix, user cannot create two identical route with only metric
> difference. It is allowed by kernel, but I don't see any real usage
> on this.
> 
> 
> Hi Dominik,
> 
> Do you need me backpoint fixes to RHEL 8.2 or/and 8.3? The workaround would
> be include a absent route for gateway in desire state also.

Germano, what is your point of view?

Reproducer:


echo '
---
interfaces:
  - name: eth1
    type: ethernet
    state: up
    ipv4:
      address:
      - ip: 192.0.2.251
        prefix-length: 24
      dhcp: false
      enabled: true

routes:
  config:
  - destination: 0.0.0.0/0
    next-hop-address: 192.0.2.1
    next-hop-interface: eth1' | sudo nmstatectl set -


echo '
---
interfaces:
  - name: eth1
    type: ethernet
    state: up
    ipv4:
      address:
      - ip: 192.0.2.251
        prefix-length: 24
      dhcp: false
      enabled: true

routes:
  config:
  - destination: 0.0.0.0/0
    next-hop-address: 192.0.2.1
    next-hop-interface: eth1
    metric: 102' | sudo nmstatectl set -

(In reply to Dominik Holler from comment #11)
> (In reply to Gris Ge from comment #10)
> > Changing to nmstate bug.
> > 
> > With nmstate-0.3 and nmstate-0.2, Nmstate will raise exception regarding two
> > gateways as nmstate think metric difference means different route.
> > 
> > With nmstate-0.4 and nmstate-1.0, Nmstata will not raise exception, but
> > system will got two gateway each with their own metrics.
> > 
> > My intention on fix is: ignore metric difference.
> > 
> > Impact on this fix, user cannot create two identical route with only metric
> > difference. It is allowed by kernel, but I don't see any real usage
> > on this.
> > 
> > 
> > Hi Dominik,
> > 
> > Do you need me backpoint fixes to RHEL 8.2 or/and 8.3? The workaround would
> > be include a absent route for gateway in desire state also.
> 
> Germano, what is your point of view?

I agree with Gris, two identical routes with only metric difference doesn't
have much use, they would need to have a difference in addition to he metric
to be considered different routes, so that both would be installed on the FIB.

Ignoring the metric difference and installing a single route makes sense to me,
as long as it logs somewhere that its consolidating 2 routes into one because
they are the same. Another thing would be to continue to fail, but the error
clearer, saying 2 routes with same metric were configured instead of duplicate
gateway.

From the RHV side it is easy to get the system into a correct state and we have
KCS. Unless the user configures routes with metrics (very unlikely) it won't
get into such state. Based on this I'd say there is no need to backport to 8.2.
If the patch applies nicely to 8.3 then why not, but if it requires those kind
of downstream only patches I'd say it's not needed and can wait for a rebase,
assuming its not too far away.

Thanks!

Adding to the above, and 8.4 is also fine, this wouldn't qualify for Z-Stream
request from CEE/GSS unless we find a much bigger problem caused by this.

In RHEL 8.4, nmstate allows multiple default gateways, so in this use case, user will have two gateways with only metric difference. It does not cause any problem just the state user will get if they use customized route metric before RHV setup the network.

Since linux kernel allows user to have same routes with different metric, let's not break any potential use case on that in nmstate and keep the old kernel behavior.

A better approach for RHV would be detecting this route metric and include it when applying via nmstate.

Since RHEL 8.3 fix is not required. After talking with RHV engineers, we agree to change this bug to ON_QA as nmstate-1.0.0-0.1.el8 already support multiple gateways.

Thanks!

Verified with versions:
nmstate-1.0.0-1.el8.noarch
nispor-1.0.1-2.el8.x86_64
NetworkManager-1.30.0-0.4.el8.x86_64
DISTRO=RHEL-8.4.0-20201217.d.2
Linux hpe-dl380pgen8-02-vm-13.hpe2.lab.eng.bos.redhat.com 4.18.0-262.el8.dt3.x86_64 #1 SMP Tue Dec 15 04:28:42 EST 2020 x86_64 x86_64 x86_64 GNU/LinuxG

Recent changes in nmstat-1.0.2. In order to support routes configured by iproute2, we have to treat different metric the same route.
So, if multiple same routes(only metric is different) are committed via nmstate, now only one route will be preserved(chosen randomly)

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (nmstate bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2021:1748