I would say that the code in vdsm/vdsm/network/configurators/ifcfg.py requires some error handling that doesn't propagate to vdsm daemon:

def _exec_ifup(iface_name, cgroup=dhclient.DHCLIENT_CGROUP):
    """Bring up an interface"""
    cmd = [constants.EXT_IFUP, iface_name]

    if cgroup is not None:
        cmd = cmdutils.systemd_run(cmd, scope=True, slice=cgroup)

    rc, out, err = utils.execCmd(cmd, raw=False)

    if rc != 0:
        # In /etc/sysconfig/network-scripts/ifup* the last line usually
        # contains the error reason.
        raise ConfigNetworkError(ERR_FAILED_IFUP, out[-1] if out else '')

This would be a less of an issue in 3.5.4, as in the typical case, Vdsm would not restore networks on its own, but rather expect to find them up by means of ONBOOT=yes.

(In reply to Dan Kenigsberg from comment #2)
> This would be a less of an issue in 3.5.4, as in the typical case, Vdsm
> would not restore networks on its own, but rather expect to find them up by
> means of ONBOOT=yes.

assuming a worse case where VDSM indeed needs to restore at least one network there are tew aspects to consider in case restoration fails:

1. Since there is a single API call to restore _all_ needed networks, this call
   either fails or succeeds. Although happily this happen transactionally, one
   network failure fails the whole restoration.
   This might be addresses by changing setupNetworks for the restoration case.
   This verb already knows if it was called during restoration or not 
   (_inRollback).
2. Network restoration takes place in a oneShot dependecy called vdsm-network.
   If this service fails as a result of network restoration failure, VDSM will
   not start.

as the first issue is worth a discussion on how to fix, the second makes sense, and 'fixing' he first will fix this scenario.

We could try to call setupNetworks from the restoration script separately (one net/bond at a time) and be ready accept failures.

However, if the management network happens to fail (VDSM does not know which is it), the host will still be unmanageable.

Also, bond creation failures might cause subsequent network creation failures.

Verified on - vdsm-4.19.2-2.el7ev.x86_64 and 4.1.0.2-0.1.el7

Now, when one or some networks failed to be restored, vdsm continue even if one failed and vdsmd is running in the end. 

restore-net::ERROR::2017-01-23 13:54:49,110::__init__::51::root::(__exit__) Failed rollback transaction last known good network.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch.py", line 153, in _setup_legacy
    bondings, _netinfo)
  File "/usr/lib/python2.7/site-packages/vdsm/network/legacy_switch.py", line 464, in add_missing_networks
    _netinfo=_netinfo, **attrs)
  File "/usr/lib/python2.7/site-packages/vdsm/network/legacy_switch.py", line 182, in wrapped
    return func(network, configurator, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/network/legacy_switch.py", line 243, in _add_network
    net_ent_to_configure.configure(**options)
  File "/usr/lib/python2.7/site-packages/vdsm/network/models.py", line 187, in configure
    self.configurator.configureBridge(self, **opts)
  File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 116, in configureBridge
    _ifup(bridge)
  File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 927, in _ifup
    _exec_ifup(iface, cgroup)
  File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 884, in _exec_ifup
    _exec_ifup_by_name(iface.name, cgroup)
  File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 870, in _exec_ifup_by_name
    raise ConfigNetworkError(ERR_FAILED_IFUP, out[-1] if out else '')
ConfigNetworkError: (29, '')
restore-net::ERROR::2017-01-23 13:54:49,111::vdsm-restore-net-config::162::root::(_greedy_setup_nets) Failed to setup n11
Traceback (most recent call last):
  File "/usr/share/vdsm/vdsm-restore-net-config", line 160, in _greedy_setup_nets
    {'connectivityCheck': False, '_inRollback': True})
  File "/usr/lib/python2.7/site-packages/vdsm/network/api.py", line 261, in setupNetworks
    _setup_networks(networks, bondings, options)
  File "/usr/lib/python2.7/site-packages/vdsm/network/api.py", line 282, in _setup_networks
    netswitch.setup(networks, bondings, options, in_rollback)
  File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch.py", line 132, in setup
    _setup_legacy(legacy_nets, legacy_bonds, options, in_rollback)
  File "/usr/lib/python2.7/site-packages/vdsm/network/netswitch.py", line 153, in _setup_legacy
    bondings, _netinfo)
  File "/usr/lib/python2.7/site-packages/vdsm/network/legacy_switch.py", line 464, in add_missing_networks
    _netinfo=_netinfo, **attrs)
  File "/usr/lib/python2.7/site-packages/vdsm/network/legacy_switch.py", line 182, in wrapped
    return func(network, configurator, **kwargs)
  File "/usr/lib/python2.7/site-packages/vdsm/network/legacy_switch.py", line 243, in _add_network
    net_ent_to_configure.configure(**options)
  File "/usr/lib/python2.7/site-packages/vdsm/network/models.py", line 187, in configure
    self.configurator.configureBridge(self, **opts)
  File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 116, in configureBridge
    _ifup(bridge)
  File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 927, in _ifup
    _exec_ifup(iface, cgroup)
  File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 884, in _exec_ifup
    _exec_ifup_by_name(iface.name, cgroup)
  File "/usr/lib/python2.7/site-packages/vdsm/network/configurators/ifcfg.py", line 870, in _exec_ifup_by_name
    raise ConfigNetworkError(ERR_FAILED_IFUP, out[-1] if out else '')
ConfigNetworkError: (29, '')
restore-net::INFO::2017-01-23 13:54:49,112::vdsm-restore-net-config::481::root::(restore) restoration completed successfully.