+++ This bug was initially created as a clone of Bug #984028 +++

Description of problem:
If logical network which had configured default gateway is removed, the host has no default gateway

Version-Release number of selected component (if applicable):
oVirt Engine Version: 3.3.0-0.2.master.20130710172219.git240a5ed.fc18 
vdsm-4.11.0-123.gitdd3f6ca.fc18

How reproducible:
100%

Steps to Reproduce:
1. Create a logical network (NET1), attach it to cluster and host
2. Hosts -> your host ->  Network Interfaces -> setup Host networks
3. Attach NET1 to host, configure static IP, netmask , gateway (apply changes)
4. remove NET1 from host via setup Host networks (apply changes)

Actual results:
no default route on host

Expected results:
default route pointing towards ovirtmgmt gateway

Additional info:

before adding logical network
[root@dell-r210ii-07 ~] # IP are
default via 10.34.66.254 dev ovirtmgmt 
10.34.66.0/24 dev ovirtmgmt  proto kernel  scope link  src 10.34.66.71 
169.254.0.0/16 dev em2  scope link  metric 1003 
169.254.0.0/16 dev ovirtmgmt  scope link  metric 1012 


after adding logical network
[root@dell-r210ii-07 ~]# ip r
default via 10.34.67.30 dev net_67_0 
10.34.66.0/24 dev ovirtmgmt  proto kernel  scope link  src 10.34.66.71 
10.34.67.0/27 dev net_67_0  proto kernel  scope link  src 10.34.67.7 
169.254.0.0/16 dev ovirtmgmt  scope link  metric 1012 
169.254.0.0/16 dev net_67_0  scope link  metric 1013 

logical network removed
[root@dell-r210ii-07 ~]# ip r
10.34.66.0/24 dev ovirtmgmt  proto kernel  scope link  src 10.34.66.71 
169.254.0.0/16 dev em2  scope link  metric 1003 
169.254.0.0/16 dev ovirtmgmt  scope link  metric 1012 


Thread-46::DEBUG::2013-07-12 16:14:40,427::fileSD::238::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n470 bytes (470 B) copied, 0.000396517 s, 1.2 MB/s\n'; <rc> = 0
Thread-2100::DEBUG::2013-07-12 16:14:49,713::BindingXMLRPC::946::vds::(wrapper) client [10.34.63.66]::call setupNetworks with ({'net_67_0': {'remove': 'true'}}, {}, {'connectivityCheck': 'true', 'connectivityTimeout': 120}) {}
Thread-2101::DEBUG::2013-07-12 16:14:49,715::BindingXMLRPC::946::vds::(wrapper) client [10.34.63.66]::call ping with () {}
Thread-2101::DEBUG::2013-07-12 16:14:49,715::BindingXMLRPC::953::vds::(wrapper) return ping with {'status': {'message': 'Done', 'code': 0}}
Thread-2102::DEBUG::2013-07-12 16:14:50,220::BindingXMLRPC::946::vds::(wrapper) client [10.34.63.66]::call ping with () {}
Thread-2102::DEBUG::2013-07-12 16:14:50,220::BindingXMLRPC::953::vds::(wrapper) return ping with {'status': {'message': 'Done', 'code': 0}}
Thread-46::DEBUG::2013-07-12 16:14:50,438::fileSD::238::Storage.Misc.excCmd::(getReadDelay) '/usr/bin/dd iflag=direct if=/rhev/data-center/mnt/10.34.63.204:_mnt_export_nfs_130_nfs03/11a8781d-28c7-410d-a1ff-3ea3474ed986/dom_md/metadata bs=4096 count=1' (cwd None)

--- Additional comment from Assaf Muller on 2013-07-14 11:00:17 EDT ---

This bug existed in the product since its inception. For example on a 3.1/3.2 setup, if a secondary (non-ovirtmgmt) network got a gateway through DHCP and was then detached, the host was left without a default gateway in its main routing table.

However, the multiple gateways feature introduced in 3.3 makes this bug more apparent.

One proposed solution is: During setupNetworks the engine should mark the management network, or the network which has the host's gateway which should be kept in the main routing table. Then we can make sure that the host will keep that network's gateway in its main table.

Another solution would be for VDSM internally to mark the management network according to its name, and keep that network's gateway in the host's main routing table.

Keep in mind that in case that the host is configured properly (using the multiple gateways feature), then the feature fixed half of the bug. Before 3.3, if the secondary network had a gateway through DHCP and was then removed, then the gateway was removed from the main routing table and the host would be completely unreachable from outside of its other subnets. Now, with 3.3, the host responds to pings to its other NICs (Even from outside of its subnets) but traffic originating FROM the host to destinations outside of its subnets will be dropped at the host.

--- Additional comment from GenadiC on 2013-07-31 06:59:06 EDT ---

Another scenario for the problem:
1) Put network red on the host (which is SPM) with SN (ip: 1.1.1.1, dg: 1.1.1.254)
2) Put the Host in Maintenance
3) Activate Host
Result: Host can't connect to the storage as it tries to go through 1.1.1.254