Could you provide more background on your use case and motivation?

You can implement your tweak by setting up a /usr/libexec/vdsm/hooks/before_network_setup/defroute.py with something like the (untested) code below:

def main():
    setup_nets_config = hooking.read_json()
    for net, attrs in setup_nets_config['request']['networks'].items():
        attrs['defaultRoute'] = net == '10_10_60_0'
    hooking.write_json(setup_nets_config)

this snipet would remove defaultRoute of the management network and pass it on to '10_10_60_0'. This could be further extended by letting the user choose the default route-bearing network by means of a custom property. Please report here if it works.

Note that with a hook like this, nothing in Engine protects you from removing the default route from the host.

I think that VDSM should at least not set DEFROUTE=YES on interfaces where there's no GATEWAY definition. Ideally there should be a checkbox setting the default route on an interface out of several ones containing gateways.

The following workaround can be used to set the default route to a non-management network in RHEV 3.5:

1) ensure networking is enabled in runlevel 3:
# chkconfig --list network
2) edit  /etc/vdsm/vdsm.conf and set "net_persistence = ifcfg"
3) restart VDSM
# service vdsmd restart

Now VDSM will not regenerate ifcfg files unless there are changes pending (default behavior before 3.5)

4) manually edit ifcfg files to set DEFROUTE=YES on the desired interface
5) reboot the host. The files should be preserved and manually overriden DEFROUTE parameter should be on the desired interface

Thank you very much Dan and Evgheni.  We will try these fixes.

Best
Doug

(In reply to Evgheni Dereveanchin from comment #2)
> 
> As there's no gateway set on "rhevm" I don't see a point in setting
> DEFROUTE=YES on this interface.

This is indeed pointless, but I also think it is harmless. Do you think otherwise?

Dan, I think the fact that DEFROUTE=YES is put on the wrong interface is the reason for this bugzilla.

If there's a 0.0.0.0/0 route on a secondary interface (let's say Display network) and no such route on the "rhevm" bridge, VDSM will still set DEFROUTE=YES on the ladder, breaking traffic originated from the hypervisor.

Is it possible to design some logic into VDSM not to add DEFROUTE=YES to "rhevm" bridge if that one does not have a gateway IP defined?

It would be impolite for Vdsm to ingnore Engine's explicit request for setting defaultRoute=True on the management network. But this can be hacked by a hook such as the one suggested in comment 1. Would you try it?

Dan,

We've been busy working on other issues in RHEV but I will try the steps suggested in comment 3.

I am not comfortable editing vdsm hooks at this time.

Thanks
Doug

Dan, can we make the Engine stop requesting defaultRoute=True when the rhevm bridge does not have a gateway defined in the DB? It seems that this is what's happening at the moment.

Is there any ETA for this?

While the suggested workaround in comment 3 still works, it occasionally leads to network-related errors in oVirt 3.6 ("Network ovirtmgmt is not attached to any interface").

DEFROUTE is adding to the main routing table a default route with the gateway as the next hope.
As VDSM uses a routing table per network, the main table is only relevant for host IP addresses that are not managed by VDSM.
Any IP address that have been set through a network setup, will not pass the main routing table and therefore not affected by this DEFROUTE.

So I see no specific need to do any hacking in the ifcfg files.

If I missed something, please let me know.

The missing point is that GATEWAY is optional; not all networks have it. Those that don't have it, are routed via the default route which is on the mgmt network; if ovirtmgmt does not have a GATEWAY defined, we end up with no routing at all.

How about solving this via a new network role of "Default Route"? Upon installation/upgrade, the management network would have it. But users would be able to set it to another cluster network. The "default route" would send defaultRoute=True to Vdsm which would end up as DEFROUTE=yes in ifcfg.

Would that work for you, Douglas?

(In reply to Dan Kenigsberg from comment #17)
> The missing point is that GATEWAY is optional; not all networks have it.
> Those that don't have it, are routed via the default route which is on the
> mgmt network; if ovirtmgmt does not have a GATEWAY defined, we end up with
> no routing at all.

Yes, I missed that point, but it does not limit the user.
If the user has a specific need, he can set it on a per network base explicitly.
If the default route/gateway is not important, he will not set it and that is it.

It seems to me that this is solvable by existing means using a proper configuration.

IMO the definition of what is a Default Route, is confusing and not intuitive (compared to the gateway definition), having a check-box/flag for it on configuration is a step towards complexity.
I would prefer to avoid it if possible.

Yesterday Dan asked me to give the proposed solution from comment #1 a try.

Scenario (these is the Config for the Host in Administration Portal)
- 2 Networks
  ovirtmgmt (192.168.0.0/24,   directly connected, not routed)
  storage3  (192.168.120.0/24, directly connected, gateway 192.168.120.1)

So this is what happens originally, when the ovirtmgmt network has no gateway defined and the storage3 has.

# ip route show | grep "default via"
[empty]
# ip route show table 3232266360 | grep "default via"
default via 192.168.120.1 dev eth4.500

This means that normally, any AF_INET socket will not hit that routing entry in the FIB during ARP resolution as there is no 'ip rule' to throw everything by default to that table 3232266360 (can also be done via systemcall from what I remember). The point is that routing table with the default route for storage3 is not the main one. Therefore "set non-mgmt network as Default Route" doesn't work.

Next, as per Dan suggestion, I installed the following hook in the host.
Note: had to modify comment #1 it quite a bit and it still has 1 bug.

#!/bin/env python

import traceback
import hooking
from vdsm import utils
from vdsm.network.netconfpersistence import RunningConfig

def main():
    setup_nets_config = hooking.read_json()
    for net, attrs in setup_nets_config['request']['networks'].items():
        df = attrs.get('custom', {}).get('default_route')
        if (df):
            hooking.log("INFO: Setting network %s gateway as default "
                            "route in main routing table" % (net))
            attrs['defaultRoute'] = True
        else:
            attrs['defaultRoute'] = False
    hooking.write_json(setup_nets_config)

if __name__ == '__main__':
    try:
        main()
    except:
        hooking.exit_hook(traceback.format_exc())

I wanted to use a custom property in order to play with it in my env via Administration Portal without having to edit hooks by hand.

So in the engine, I did this:
# engine-config -s 'UserDefinedNetworkCustomProperties=default_route=.*' --cver='4.0'

The hook looks for this custom property for the host network called "default_route" (which is NOT the original DefaultRoute attribute).

Then, when configuring the network for the host in the GUI, I set this 'default_route' custom property to True.

The hook, once this is received, set all networks (non custom) defaultRoute to False, except the one that has the custom default_route set.

Note the custom property default_route set to True for storage3 and defaultRoute set to False for ovirtmgmt but true for storage3.

MainProcess|jsonrpc.Executor/3::INFO::2016-11-04 00:23:02,262::netconfpersistence::132::root::(save) Saved new config RunningConfig({'ovirtmgmt': {u'ipv6autoconf': False, u'ipaddr': u'192.168.0.120', u'bonding': u'bond0', u'mtu': 1350, u'switch': u'legacy', u'dhcpv6': False, u'stp': False, u'bridged': True, u'netmask': u'255.255.255.0', u'defaultRoute': False}, u'storage3': {u'ipv6autoconf': False, u'nic': u'eth4', u'vlan': 500, u'ipaddr': u'192.168.120.120', u'switch': u'legacy', u'mtu': 1350, u'netmask': u'255.255.255.0', u'dhcpv6': False, u'bridged': False, u'custom': {u'default_route': u'True'}, u'gateway': u'192.168.120.1', u'defaultRoute': True}}, ... to /var/run/vdsm/netconf/nets/ and /var/run/vdsm/netconf/bonds/

Results:
# ip route show table main | grep "default via"
default via 192.168.120.1 dev eth4.500 

\o/ \o/
SUCCESS!! The gateway of storage3 was now added to the main table. So after this a non-mgmt network is set as default route.

TODO:
1) If the solution for this BZ will head this way, we would like a logical-network wide custom property setting, as adding default_route = True for each single host is tedious. We should have custom properties under the Networks tab as well.
2) The default route is still being installed in the "alternative" table and the ip rules are also installed. I don't see any point of doing this if the gateway is now in the main table. Need to clean this up. There is no need to direct traffic arriving on storage3 to a different RT if it contains the same default route as the main one. Example:

# ip route show table all | grep "default via"
default via 192.168.120.1 dev eth4.500  table 3232266360 
default via 192.168.120.1 dev eth4.500 

# ip rule
0:	from all lookup local 
32764:	from all to 192.168.120.0/24 iif eth4.500 lookup 3232266360 
32765:	from 192.168.120.0/24 lookup 3232266360 
32766:	from all lookup main 
32767:	from all lookup default 

Hopefully this helps.

Cheers,
Germano

Thanks Dan for creating a vdsm patch for this, so no hook is necessary.

https://gerrit.ovirt.org/#/c/66127/

Here are the configurations:

engine-config -s CustomDeviceProperties='{type=interface;prop={default_reoute=^(true|false)$}}'

storage3     | STATIC_IP     | 192.168.120.110 | 192.168.120.1 | "default_route" : "True"

 ovirtmgmt   | DHCP          |                 |               | "default_route" : "False"

Here are the results:

# ip route show table all | grep "default via"
default via 192.168.0.254 dev ovirtmgmt  table 3232235630 
default via 192.168.120.1 dev eth4.500  table 3232266350 
default via 192.168.120.1 dev eth4.500 

# ip rule
0:	from all lookup local 
32762:	from all to 192.168.120.0/24 iif eth4.500 lookup 3232266350 
32763:	from 192.168.120.0/24 lookup 3232266350 
32764:	from all to 192.168.0.0/24 iif ovirtmgmt lookup 3232235630 
32765:	from 192.168.0.0/24 lookup 3232235630 
32766:	from all lookup main 
32767:	from all lookup default 

Looks good to me. Not sure about the usefulness of having table 3232266350, but IMO that's not related to this BZ.

The only note is that we probably want --cver='4.0' or higher when configuring the custom property, because I don't think we will backport this to 3.6 vdsm. But that's a documentation thing.

Thanks!

Got the latest stable vdsm we ship in RHV: vdsm-4.18.13-1.el7ev.x86_64
And cherry-picked these into it:
  33eabee - net: let users override Engine's default route
* f6509f5 - net: Validate nameservers setup  	
* 710e76e - net: support nameserver address with %iface tail 
  2ebb4f4 - net: Validate that default route is set for a single network

* Not the object of this test, but required in order to apply 2ebb4f4.

Initial State:
ovirtmgmt : default route
all others : no default route but with def gw set in the network config

Test 1: "Sync All Networks"
Result -> Success

Test 2: Set network storage3 as default route
Result: "Message: Error while executing action HostSetupNetworks: Illegal Network parameters" -> Success, click cancel

Test 3: Set ovirtmgmt custom property to False and storage3 to True:
Result: Routing table the same as comment #43 -> Success

Test 4: Remove ovirtmgmt custom property
Result: "Error while executing action HostSetupNetworks: Illegal Network parameters" -> Success, click cancel

Test 5: Remove the default gateway of storage3
Result: Main routing table with no default route and no warnings or errors displayed -> Fail

I don't think Test 5 is something we covered before. But with this new feature it would be nice if we warned the user that the config is inconsistent (setting a default route via a network that has no gateway).

Thanks,
Germano

> 
> Test 5: Remove the default gateway of storage3
> Result: Main routing table with no default route and no warnings or errors
> displayed -> Fail
> 
> I don't think Test 5 is something we covered before. But with this new
> feature it would be nice if we warned the user that the config is
> inconsistent (setting a default route via a network that has no gateway).
> 

We silently ignore this as it has no meaning.
At the UI we cannot validate and warn about it, as it uses custom properties.
I do not see why we should fail the setup because of this on the host side, it feels a bit of an overdo.

(In reply to Edward Haas from comment #45)
> We silently ignore this as it has no meaning.
> At the UI we cannot validate and warn about it, as it uses custom properties.
> I do not see why we should fail the setup because of this on the host side,
> it feels a bit of an overdo.

Sorry for the late response :(

Hmmm. Why no meaning?

From a networking perspective, I if set a "default_route=true" in an interface without specifying a next-hop IP address it is implicit that we would add something like this:

ip route add X.X.X.X/M dev DEV.

Implying there there is some sort of proxy-arp enabled in the link of DEV (presumably the gateway will answer ARP requests for every IP not in the LAN that it has routes to reach).

http://www.cisco.com/c/en/us/support/docs/ip/dynamic-address-allocation-resolution/13718-5.html

But AFAICS, we don't do this - we don't add that Route, and we don't fail.

I believe the correct behavior would be to add the route (proxy arp setup) or to fail.

(In reply to Germano Veit Michel from comment #46)
> 
> From a networking perspective, I if set a "default_route=true" in an
> interface without specifying a next-hop IP address it is implicit that we
> would add something like this:
> 
> ip route add X.X.X.X/M dev DEV.
> 
> Implying there there is some sort of proxy-arp enabled in the link of DEV
> (presumably the gateway will answer ARP requests for every IP not in the LAN
> that it has routes to reach).
> 
> http://www.cisco.com/c/en/us/support/docs/ip/dynamic-address-allocation-
> resolution/13718-5.html
> 
> But AFAICS, we don't do this - we don't add that Route, and we don't fail.
> 
> I believe the correct behavior would be to add the route (proxy arp setup)
> or to fail.

Using the iface on which the network is defined on as the next-hop is a valid option, but we do not support it at the moment. I would also prefer to see this explicitly set and not implicitly.

If there is no RFE for this, I think  we should open one.

IMO, until we will support the ability to control routes in general, we can keep the current logic of silently ignoring this type of misconfiguration.
But I have no strong feelings about it.
Dan, what do you think?

(In reply to Edward Haas from comment #47)
> I would also prefer to see this explicitly set and not implicitly.

Agreed.

> IMO, until we will support the ability to control routes in general, we can
> keep the current logic of silently ignoring this type of misconfiguration.
> But I have no strong feelings about it.

As an alternate solution we can document the default route custom property and explicitly say in the documentation that it requires a next hop or the route will be ignored. I would be happy to help in a Doc BZ to ensure it's properly explained.

> 
> As an alternate solution we can document the default route custom property
> and explicitly say in the documentation that it requires a next hop or the
> route will be ignored. I would be happy to help in a Doc BZ to ensure it's
> properly explained.

Sounds good to me.
Can you open a BZ for this and push it in? Do you prefer me to do it?

(In reply to Edward Haas from comment #49)
> > 
> > As an alternate solution we can document the default route custom property
> > and explicitly say in the documentation that it requires a next hop or the
> > route will be ignored. I would be happy to help in a Doc BZ to ensure it's
> > properly explained.
> 
> Sounds good to me.
> Can you open a BZ for this and push it in? Do you prefer me to do it?

Cool!

Done. BZ #1400366.

Since we are targeting this for RHV 4.2, that one should have the same target. But I am not supposed to set it, so left it blank.

Verified on - 4.2.0-0.5.master.el7

The documentation text flag should only be set after 'doc text' field is provided. Please provide the documentation text and set the flag to '?' again.

(In reply to Red Hat Bugzilla Rules Engine from comment #53)
> The documentation text flag should only be set after 'doc text' field is
> provided. Please provide the documentation text and set the flag to '?'
> again.

ylavi, it seems like a robot on rampage; unsetting a flag that was set several months ago:

 Emma Heftman 2017-03-22 11:16:31 IST
Doc Text: Feature: Ability to set via UI non-mgmt network as one, which is used to setup default route. Rea... → Feature: Ability to set via UI non-mgmt network as one, which is used to setup default route. Rea...
Red Hat Bugzilla 2017-03-22 11:16:31 IST
Flags: requires_doc_text? → requires_doc_text+

(In reply to Dan Kenigsberg from comment #54)
> (In reply to Red Hat Bugzilla Rules Engine from comment #53)
> > The documentation text flag should only be set after 'doc text' field is
> > provided. Please provide the documentation text and set the flag to '?'
> > again.
> 
> ylavi, it seems like a robot on rampage; unsetting a flag that was set
> several months ago:
> 
>  Emma Heftman 2017-03-22 11:16:31 IST
> Doc Text: Feature: Ability to set via UI non-mgmt network as one, which is
> used to setup default route. Rea... → Feature: Ability to set via UI
> non-mgmt network as one, which is used to setup default route. Rea...
> Red Hat Bugzilla 2017-03-22 11:16:31 IST
> Flags: requires_doc_text? → requires_doc_text+

Changed the rule. Hoping that it doesn't happen again.

The documentation text flag should only be set after 'doc text' field is provided. Please provide the documentation text and set the flag to '?' again.

The documentation text flag should only be set after 'doc text' field is provided. Please provide the documentation text and set the flag to '?' again.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2018:1488

BZ<2>Jira Resync