After your step (2), we know that the host is actually out of sync (though Engine does not show that): the cluster thinks that non-mgmt is the default route network, but on the host - it is still ovirtmgmt.

Could you check the following workaround: make any kind of modification to ovirtmgmt (say, change MTU), and sync the host. This should clear default_route from ovirtmgmt and let you continute to your step (3).

default route still reported in vdsm for ovirtmgmt 

"ovirtmgmt": {
            "ipv6autoconf": true, 
            "addr": "10.35.128.18", 
            "dhcpv6": false, 
            "ipv6addrs": [], 
            "mtu": "1500", 
            "dhcpv4": true, 
            "netmask": "255.255.255.0", 
            "ipv4defaultroute": true, 

"default_route_n": {
            "ipv6autoconf": false, 
            "addr": "", 
            "dhcpv6": false, 
            "ipv6addrs": [], 
            "mtu": "1500", 
            "dhcpv4": false, 
            "netmask": "", 
            "ipv4defaultroute": false,

additional info:

we have attached mgmt network, and new new_network, which is not attached. Rows in this table are ordered so mgmt network is first, new_network is second (and only this will grant proper order of execution; remember this note, this will likely cause failure in future). Logic then goes like this. First we update mgmt network, removing default route role from it. When we're removing set flag, we will use mgmt network as a replacement. So this will actually move default route from mgmt network back to mgmt network (2 db calls). Then we update new_network, which sets this role to new_network and remove it from mgmt network (another 2 db calls). All of this leads to 2 updates, but not attachment or detachment of network, which drives PropagateLabeledNetworksToClusterHostsCommand not to call setupnetworks at all. This is I believe a bug. (or not? is there case where must not call setupnetwoks in ManageNetworkCluster?) Anyway, in this case we updated db, but did not update hosts.

But lets assume that we ran setupneworks from ManageNetworkCluster. What is expected behavior? new_network is not attached, so it won't define default route on host.

Reason why this fail is, that we are jus attaching new_network, but did not touch old one, meaning, we will sent new network to vdsm and the old one will probably be unchanged.

And finally, this reproduction scenario does not work all the time. Now it works for me, even if I drop host and whole db.


related bugs?: 
• removal of network with default route role, causes system state without network of this role.

fresh env
create host
create new network
make it default route
open setupnetworks dialog, configure static ip there, attach
———

network was attached, correctly. So I believe that fixing other bugs fix this one as well.

https://gerrit.ovirt.org/#/q/status:merged+project:ovirt-engine+branch:master+topic:defaultRoute-DNS are in

Not pass. Same result.
Tested on 4.2.0-0.0.master.20170721095131.git9f5e90c.el7.centos
and vdsm-4.20.1-218.git1b7671f.el7.centos.x86_64

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/API.py", line 1409, in setupNetworks
    supervdsm.getProxy().setupNetworks(networks, bondings, options)
  File "/usr/lib/python2.7/site-packages/vdsm/supervdsm.py", line 53, in __call__
    return callMethod()
  File "/usr/lib/python2.7/site-packages/vdsm/supervdsm.py", line 51, in <lambda>
    **kwargs)
  File "<string>", line 2, in setupNetworks
  File "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in _callmethod
    raise convert_to_error(kind, result)
ConfigNetworkError: (21, 'Only a single default route network is allowed.')

Please note, that this bug isn't fixed properly. The fix for this bug affecting all relevant bugs for the default route feature.

1) You still can't attach a non-mgmt network as a default route network to the host. As engine believes that we trying to attach 2 networks with default route.
2) Engine reports the management as out-of-sync, default route on host=true, default route in the DC=false. We missing a refresh caps here as well.

When we setting a non-mgmt network with default route role, engine must take an action and send refresh caps for the management network with default route =false and sync all hosts in the cluster. With out it, all the hosts in the cluster will be out of sync and you won't be able to attach a non-mgmt network with default route to any host until the management network will be taken care by the engine. We will always fail as engine will think we trying to attach 2 networks with default route role.

*** Bug 1434087 has been marked as a duplicate of this bug. ***

Verified on - 4.2.0-0.0.master.20170917124606.gita804ef7.el7.centos

This bugzilla is included in oVirt 4.2.0 release, published on Dec 20th 2017.

Since the problem described in this bug report should be
resolved in oVirt 4.2.0 release, published on Dec 20th 2017, it has been closed with a resolution of CURRENT RELEASE.

If the solution does not work for you, please open a new bug report.